Introduction
Traditional code generation benchmarks, such as HumanEval and MBPP, have long served as the standard for evaluating Large Language Models (LLMs). However, these benchmarks often suffer from a narrow focus, typically evaluating models on standalone function-level tasks primarily in Python or Java. In contrast, real-world software engineering is a complex, multi-granular, and polyglot endeavor. Developers frequently require assistance at various structural levels—from autocompleting a single line of logic to scaffolding entire object-oriented class structures—across a diverse array of programming languages.
To bridge this gap, this paper introduces M2G-Eval, a comprehensive benchmark and training framework designed to evaluate and enhance LLMs across 18 programming languages and four distinct structural granularities: Line, Block, Function, and Class. By moving beyond simple function-level generation, M2G-Eval provides a more rigorous and realistic assessment of a model's true coding capabilities.
The M2G-Eval Framework
Multi-Granularity Parsing and Context
The researchers utilized Tree-Sitter to parse Abstract Syntax Trees (ASTs) from raw source code, allowing them to isolate specific editable units at the Line, Block, Function, and Class levels. Recognizing that code is rarely written in isolation, the framework employs BM25 to retrieve relevant cross-file context, ensuring the model has access to necessary dependencies and project-specific logic. To further assist the model, a high-capacity LLM (Qwen3-Coder) was used to generate natural language descriptions of the masked code segments, providing semantic guidance for the generation task.
Evaluation Metric: Length-Normalized Edit Similarity
Unlike execution-based benchmarks that require complex, language-specific environments, M2G-Eval utilizes Length-Normalized Edit Similarity. This metric, based on a modified Levenshtein distance, measures the syntactic proximity of the model's output to the ground-truth reference code. This approach allows for scalable evaluation across 18 different languages, including niche languages where execution environments are difficult to standardize.
Training Pipeline
The authors developed a two-stage training pipeline to optimize model performance on multi-granular tasks:
- Supervised Fine-Tuning (SFT): The base model (Qwen3-8B) is first trained on a high-quality instruction dataset.
- Reinforcement Learning (RL): The model is further refined using Group Relative Policy Optimization (GRPO). In this stage, the model is rewarded based on the edit similarity score of its generated code, directly aligning its training objective with the evaluation metric.
Datasets and Contamination Mitigation
The study utilizes two primary datasets:
- M2G-Eval-Instruct: A training set comprising approximately 17,000 high-quality tasks synthesized from 150,000 repositories sampled from The Stack v2.
- M2G-Eval Benchmark: A human-annotated test set of 1,286 instances. To ensure the integrity of the evaluation and prevent data contamination, all test instances were sourced from GitHub repositories created or updated after January 1, 2024, ensuring they were not part of the models' original pre-training data.
Key Insights
The Difficulty Gradient
The evaluation reveals a clear hierarchy of difficulty in code generation. Models perform best at the Line level, with performance decreasing through Block and Function levels, and dropping significantly at the Class level. This suggests that as the structural scope and architectural complexity increase, LLMs struggle to maintain coherence and satisfy multi-faceted constraints.
The Language Gap
The researchers observed a widening performance gap between "full-granularity" (data-rich languages like Python and Java) and "partial-granularity" (data-poor languages like Verilog or Lua). While models are relatively competent at line-level completion across most languages, their ability to handle complex structures in niche languages is markedly inferior to their performance in popular ones.
Transferable Reasoning
Interestingly, the study found strong positive correlations in model scores across nearly all language pairs. This indicates that modern LLMs are developing underlying, transferable programming logic and reasoning capabilities that transcend specific syntax. A model that excels in reasoning through a complex Java problem is statistically likely to perform well in C# or Rust as well.
Results and Performance
The authors benchmarked 30 models, including 28 state-of-the-art LLMs and two custom variants. While closed-source models like Claude-3.7-Sonnet and Gemini-2.5-Pro remain the top performers, open-source models are rapidly narrowing the gap.
Notably, the authors' M2G-Eval-Coder-RL (8B) model demonstrated exceptional efficiency. By utilizing the GRPO reinforcement learning stage, this 8B parameter model consistently outperformed its base version and rivaled the performance of significantly larger models, such as the 235B parameter Qwen3 variant, on multi-granularity tasks.
Limitations and Future Directions
While M2G-Eval provides a more holistic view of code generation, it has limitations. The reliance on syntactic similarity rather than execution-based correctness means that a model might generate code that looks correct but contains subtle logical errors or fails to compile. Additionally, the data distribution of the real world means that niche languages still lack sufficient test cases at the highest granularities.
Future work will focus on scaling execution-based environments to support multi-granularity, cross-file evaluation across a broader spectrum of languages, further refining the accuracy of AI code assessment.
Summary updated