CODEBLOCK: Learning to Supervise Code at the Right Granularity

Zhijie Deng, Ling Li, Jinlong Pang, Kaiqin Hu, Qi Xuan, Zhaowei Zhu, Jiaheng WeiarXiv 2026 · arXiv

TL;DR

Supervised fine-tuning (SFT) for code LLMs traditionally calculates loss on every single token in a response, which introduces optimization noise and computational inefficiency. While token-level filtering works well for natural language, it fragments code syntax and breaks logical dependencies. CODEBLOCK introduces a structure-aware sparse supervision framework that selects complete, data-flow-critical code blocks for training, matching or outperforming full-token SFT while supervising just 1.9% of response tokens.

What Problem Does This Paper Address?

When fine-tuning large language models, standard SFT applies a uniform cross-entropy loss to all tokens in a target response. Recent work in natural language processing has shown that not all tokens are equally valuable; supervising only high-utility tokens can maintain or even improve performance.

However, directly applying token-level masking to code fails due to a "granularity mismatch." Code semantics are not local to isolated tokens; they are jointly formed by syntactic structures and data-flow relations (like variable definitions and uses). Pointwise token selection tends to pick scattered tokens across different statements, leaving out critical context like assignment operators or closing brackets. The authors show that even with a 10% token budget, 92.3% of code items selected by standard token-level methods are structurally incomplete, which severely degrades downstream learning.

Figure 2: Comparison between previous token-level selection and CODEBLOCK. While prior methods se- lect isolated high-score tokens, CODEBLOCK preserves complete coding items with coherent syntax and data dependencies.

Methods

CODEBLOCK addresses this challenge through a structure-aware sparse supervision pipeline consisting of four main stages:

  1. Sample-Level Selection: It filters a high-quality, diverse 30K subset from a larger instruction pool using a combination of LLM-judge ratings, unit test execution, and a long-tail structural rarity score (prioritizing rare APIs or complex control flows).
  2. GCE-Based Coding Item Scoring: Code responses are partitioned into "coding items" (minimal local units expressing a coherent computation, like an assignment or API call) using Tree-sitter. It computes Generalized Cross Entropy (GCE) over core logic tokens to measure model uncertainty while tempering the influence of extreme outliers.
  3. Data-Flow Guided Item Selection: To ensure selected blocks are functionally important, CODEBLOCK builds a lightweight static data-flow graph. It scores items based on Reach (downstream influence) and Bridge (how critical the item is to connecting early definitions to final outputs). A gated priority function combines the GCE utility score with these structural data-flow signals.
  4. Sparse Fine-Tuning: During training, the full response remains available as autoregressive context so the model retains exposure to valid code structures, but the loss mask is restricted only to the top-priority code blocks and highly informative natural language tokens.

Datasets

  • Training Pool: 300K instruction-tuning examples sampled from OpenCodeInstruct, curated down to a 30K training subset.
  • Evaluation Benchmarks: Six prominent code-generation benchmarks: HumanEval, HumanEval+, MBPP, MBPP+, BigCodeBench-Hard, and BigCodeBench-Full.

Key Insights and Ideas

  • Anchors vs. Targets: High-loss individual tokens are excellent anchors for locating where a model is struggling, but they are poor targets for isolated supervision. They must be expanded into syntactically closed blocks to be effective.
  • Decoupling Context from Loss: You do not need to compute gradients on a token for the model to learn from its presence. Keeping unselected tokens in the autoregressive context preserves structural awareness while avoiding the noise of dense supervision.
  • Uncertainty Alone is Insufficient: Traditional token selection relies purely on loss/uncertainty. CODEBLOCK demonstrates that incorporating data-flow metrics ensures the model prioritizes learning code blocks that actually propagate critical program dependencies.

Key Results

  • Superior Performance-Efficiency Trade-off: CODEBLOCK consistently matched or outperformed full-token SFT across multiple base models (including Qwen2.5-Coder, Seed-Coder, and OpenCoder) while using only 1.9% of effective supervised tokens.
  • Quantitative Gains: On Qwen2.5-Coder-1.5B-Instruct, CODEBLOCK achieved an average score of 54.6 across benchmarks, outperforming full-token SFT (52.7) and beating competitive sparse baselines like Token Cleaning (49.1) and CLAM (51.8).
  • Significant Speedups: Total wall-clock runtime (including preprocessing and training) was cut down to just 16.3% of full-token SFT, dropping from 255 minutes to 41.5 minutes on a single NVIDIA A800 GPU.
Figure 1: Performance–efficiency trade-off on Qwen2.5- Coder-1.5B-Instruct. CODEBLOCK achieves the highest average performance with only 1.9% effective super- vised tokens, showing a better trade-off than full-token SFT and sparse-selection baselines.

Limitations and Future Work

The primary limitation is CODEBLOCK’s reliance on static data-flow analysis to build its dependency graphs. While static analysis is highly scalable and lightweight, it cannot fully capture complex runtime behaviors such as dynamic dispatch, pointer aliasing, object mutation, or input-dependent execution paths.

As a result, the structural signals (Reach and Bridge) remain approximations. The authors suggest that future work could integrate dynamic execution traces or more rigorous program analysis frameworks to construct richer, more precise dependency graphs.

Summary updated

Abstract

Supervised fine-tuning of code LLMs typically applies uniform cross-entropy loss to all response tokens, implicitly assuming that every token provides equally useful learning signal. Recent token-level selection methods challenge this assumption in natural-language SFT by supervising only high-value tokens. However, directly transferring token-level masking to code can break syntactically and semantically coherent program units, because code depends on structural completeness and definition-use relations. We therefore propose CodeBlock, a structure-aware sparse supervision framework that selects structure-complete code evidence rather than isolated tokens. CodeBlock first selects high-quality instruction-response pairs, then partitions code responses into syntactically coherent coding items, estimates their utility by aggregating generalized cross-entropy over core logic tokens, and reranks them with data-flow reach and bridge signals to prioritize blocks that propagate or connect important program dependencies. During training, the full response remains available as context, while loss is applied only to selected code items and informative natural-language tokens. Experiments on six code-generation benchmarks show that CodeBlock achieves stronger average pass@1 than full-token SFT and competitive selection baselines, while using only 1.9% of supervised response tokens.
BibTeX
@misc{deng2026codeblock,
  author = {Zhijie Deng and Ling Li and Jinlong Pang and Kaiqin Hu and Qi Xuan and Zhaowei Zhu and Jiaheng Wei},
  title = {{CODEBLOCK: Learning to Supervise Code at the Right Granularity}},
  year = {2026},
  eprint = {2606.18286},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.18286},
}
Figure