M2G-Eval: Enhancing and Evaluating Multi-granularity Multilingual Code Generation

Fanglin Xu, Wei Zhang, Jian Yang, Guo Chen, Aishan Liu, Zhoujun Li, Xianglong Liu, Bryan DaiarXiv 2025 · DOI · arXiv

Introduction

Traditional code generation benchmarks, such as HumanEval and MBPP, have long served as the standard for evaluating Large Language Models (LLMs). However, these benchmarks often suffer from a narrow focus, typically evaluating models on standalone function-level tasks primarily in Python or Java. In contrast, real-world software engineering is a complex, multi-granular, and polyglot endeavor. Developers frequently require assistance at various structural levels—from autocompleting a single line of logic to scaffolding entire object-oriented class structures—across a diverse array of programming languages.

To bridge this gap, this paper introduces M2G-Eval, a comprehensive benchmark and training framework designed to evaluate and enhance LLMs across 18 programming languages and four distinct structural granularities: Line, Block, Function, and Class. By moving beyond simple function-level generation, M2G-Eval provides a more rigorous and realistic assessment of a model's true coding capabilities.

The M2G-Eval Framework

Multi-Granularity Parsing and Context

The researchers utilized Tree-Sitter to parse Abstract Syntax Trees (ASTs) from raw source code, allowing them to isolate specific editable units at the Line, Block, Function, and Class levels. Recognizing that code is rarely written in isolation, the framework employs BM25 to retrieve relevant cross-file context, ensuring the model has access to necessary dependencies and project-specific logic. To further assist the model, a high-capacity LLM (Qwen3-Coder) was used to generate natural language descriptions of the masked code segments, providing semantic guidance for the generation task.

Evaluation Metric: Length-Normalized Edit Similarity

Unlike execution-based benchmarks that require complex, language-specific environments, M2G-Eval utilizes Length-Normalized Edit Similarity. This metric, based on a modified Levenshtein distance, measures the syntactic proximity of the model's output to the ground-truth reference code. This approach allows for scalable evaluation across 18 different languages, including niche languages where execution environments are difficult to standardize.

Training Pipeline

The authors developed a two-stage training pipeline to optimize model performance on multi-granular tasks:

  1. Supervised Fine-Tuning (SFT): The base model (Qwen3-8B) is first trained on a high-quality instruction dataset.
  2. Reinforcement Learning (RL): The model is further refined using Group Relative Policy Optimization (GRPO). In this stage, the model is rewarded based on the edit similarity score of its generated code, directly aligning its training objective with the evaluation metric.

Datasets and Contamination Mitigation

The study utilizes two primary datasets:

  • M2G-Eval-Instruct: A training set comprising approximately 17,000 high-quality tasks synthesized from 150,000 repositories sampled from The Stack v2.
  • M2G-Eval Benchmark: A human-annotated test set of 1,286 instances. To ensure the integrity of the evaluation and prevent data contamination, all test instances were sourced from GitHub repositories created or updated after January 1, 2024, ensuring they were not part of the models' original pre-training data.

Key Insights

The Difficulty Gradient

The evaluation reveals a clear hierarchy of difficulty in code generation. Models perform best at the Line level, with performance decreasing through Block and Function levels, and dropping significantly at the Class level. This suggests that as the structural scope and architectural complexity increase, LLMs struggle to maintain coherence and satisfy multi-faceted constraints.

The Language Gap

The researchers observed a widening performance gap between "full-granularity" (data-rich languages like Python and Java) and "partial-granularity" (data-poor languages like Verilog or Lua). While models are relatively competent at line-level completion across most languages, their ability to handle complex structures in niche languages is markedly inferior to their performance in popular ones.

Transferable Reasoning

Interestingly, the study found strong positive correlations in model scores across nearly all language pairs. This indicates that modern LLMs are developing underlying, transferable programming logic and reasoning capabilities that transcend specific syntax. A model that excels in reasoning through a complex Java problem is statistically likely to perform well in C# or Rust as well.

Results and Performance

The authors benchmarked 30 models, including 28 state-of-the-art LLMs and two custom variants. While closed-source models like Claude-3.7-Sonnet and Gemini-2.5-Pro remain the top performers, open-source models are rapidly narrowing the gap.

Notably, the authors' M2G-Eval-Coder-RL (8B) model demonstrated exceptional efficiency. By utilizing the GRPO reinforcement learning stage, this 8B parameter model consistently outperformed its base version and rivaled the performance of significantly larger models, such as the 235B parameter Qwen3 variant, on multi-granularity tasks.

Limitations and Future Directions

While M2G-Eval provides a more holistic view of code generation, it has limitations. The reliance on syntactic similarity rather than execution-based correctness means that a model might generate code that looks correct but contains subtle logical errors or fails to compile. Additionally, the data distribution of the real world means that niche languages still lack sufficient test cases at the highest granularities.

Future work will focus on scaling execution-based environments to support multi-granularity, cross-file evaluation across a broader spectrum of languages, further refining the accuracy of AI code assessment.

Summary updated

Abstract

The rapid advancement of code large language models (LLMs) has sparked significant research interest in systematically evaluating their code generation capabilities, yet existing benchmarks predominantly assess models at a single structural granularity and focus on limited programming languages, obscuring fine-grained capability variations across different code scopes and multilingual scenarios. We introduce M2G-Eval, a multi-granularity, multilingual framework for evaluating code generation in large language models (LLMs) across four levels: Class, Function, Block, and Line. Spanning 18 programming languages, M2G-Eval includes 17K+ training tasks and 1,286 human-annotated, contamination-controlled test instances. We develop M2G-Eval-Coder models by training Qwen3-8B with supervised fine-tuning and Group Relative Policy Optimization. Evaluating 30 models (28 state-of-the-art LLMs plus our two M2G-Eval-Coder variants) reveals three main findings: (1) an apparent difficulty hierarchy, with Line-level tasks easiest and Class-level most challenging; (2) widening performance gaps between full- and partial-granularity languages as task complexity increases; and (3) strong cross-language correlations, suggesting that models learn transferable programming concepts. M2G-Eval enables fine-grained diagnosis of code generation capabilities and highlights persistent challenges in synthesizing complex, long-form code.
BibTeX
@misc{xu2025m,
  author = {Fanglin Xu and Wei Zhang and Jian Yang and Guo Chen and Aishan Liu and Zhoujun Li and Xianglong Liu and Bryan Dai},
  title = {{M2G-Eval: Enhancing and Evaluating Multi-granularity Multilingual Code Generation}},
  year = {2025},
  eprint = {2512.22628},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.22628},
}
Figure