TL;DR
Supervised fine-tuning (SFT) for code LLMs traditionally calculates loss on every single token in a response, which introduces optimization noise and computational inefficiency. While token-level filtering works well for natural language, it fragments code syntax and breaks logical dependencies. CODEBLOCK introduces a structure-aware sparse supervision framework that selects complete, data-flow-critical code blocks for training, matching or outperforming full-token SFT while supervising just 1.9% of response tokens.
What Problem Does This Paper Address?
When fine-tuning large language models, standard SFT applies a uniform cross-entropy loss to all tokens in a target response. Recent work in natural language processing has shown that not all tokens are equally valuable; supervising only high-utility tokens can maintain or even improve performance.
However, directly applying token-level masking to code fails due to a "granularity mismatch." Code semantics are not local to isolated tokens; they are jointly formed by syntactic structures and data-flow relations (like variable definitions and uses). Pointwise token selection tends to pick scattered tokens across different statements, leaving out critical context like assignment operators or closing brackets. The authors show that even with a 10% token budget, 92.3% of code items selected by standard token-level methods are structurally incomplete, which severely degrades downstream learning.
Methods
CODEBLOCK addresses this challenge through a structure-aware sparse supervision pipeline consisting of four main stages:
- Sample-Level Selection: It filters a high-quality, diverse 30K subset from a larger instruction pool using a combination of LLM-judge ratings, unit test execution, and a long-tail structural rarity score (prioritizing rare APIs or complex control flows).
- GCE-Based Coding Item Scoring: Code responses are partitioned into "coding items" (minimal local units expressing a coherent computation, like an assignment or API call) using Tree-sitter. It computes Generalized Cross Entropy (GCE) over core logic tokens to measure model uncertainty while tempering the influence of extreme outliers.
- Data-Flow Guided Item Selection: To ensure selected blocks are functionally important, CODEBLOCK builds a lightweight static data-flow graph. It scores items based on Reach (downstream influence) and Bridge (how critical the item is to connecting early definitions to final outputs). A gated priority function combines the GCE utility score with these structural data-flow signals.
- Sparse Fine-Tuning: During training, the full response remains available as autoregressive context so the model retains exposure to valid code structures, but the loss mask is restricted only to the top-priority code blocks and highly informative natural language tokens.
Datasets
- Training Pool: 300K instruction-tuning examples sampled from OpenCodeInstruct, curated down to a 30K training subset.
- Evaluation Benchmarks: Six prominent code-generation benchmarks: HumanEval, HumanEval+, MBPP, MBPP+, BigCodeBench-Hard, and BigCodeBench-Full.
Key Insights and Ideas
- Anchors vs. Targets: High-loss individual tokens are excellent anchors for locating where a model is struggling, but they are poor targets for isolated supervision. They must be expanded into syntactically closed blocks to be effective.
- Decoupling Context from Loss: You do not need to compute gradients on a token for the model to learn from its presence. Keeping unselected tokens in the autoregressive context preserves structural awareness while avoiding the noise of dense supervision.
- Uncertainty Alone is Insufficient: Traditional token selection relies purely on loss/uncertainty. CODEBLOCK demonstrates that incorporating data-flow metrics ensures the model prioritizes learning code blocks that actually propagate critical program dependencies.
Key Results
- Superior Performance-Efficiency Trade-off: CODEBLOCK consistently matched or outperformed full-token SFT across multiple base models (including Qwen2.5-Coder, Seed-Coder, and OpenCoder) while using only 1.9% of effective supervised tokens.
- Quantitative Gains: On Qwen2.5-Coder-1.5B-Instruct, CODEBLOCK achieved an average score of 54.6 across benchmarks, outperforming full-token SFT (52.7) and beating competitive sparse baselines like Token Cleaning (49.1) and CLAM (51.8).
- Significant Speedups: Total wall-clock runtime (including preprocessing and training) was cut down to just 16.3% of full-token SFT, dropping from 255 minutes to 41.5 minutes on a single NVIDIA A800 GPU.
Limitations and Future Work
The primary limitation is CODEBLOCK’s reliance on static data-flow analysis to build its dependency graphs. While static analysis is highly scalable and lightweight, it cannot fully capture complex runtime behaviors such as dynamic dispatch, pointer aliasing, object mutation, or input-dependent execution paths.
As a result, the structural signals (Reach and Bridge) remain approximations. The authors suggest that future work could integrate dynamic execution traces or more rigorous program analysis frameworks to construct richer, more precise dependency graphs.
Summary updated