Code summarizers learn from the docstrings that developers wrote next to their functions, and many of those docstrings are wrong. Some describe behavior the code lost in a refactor. Some were copy-pasted from a sibling function. Some make claims about a library the function merely calls. The main tool for training on noisy targets is loss truncation, which drops the training examples or tokens the model finds most surprising. It rests on an assumption borrowed from how annotators are usually simulated: errors are idiosyncratic and independent, so a wrong target looks unusual to the model. Real docstring errors are not independent. The same stale phrase recurs across clones, and the same outdated wording sits in an inline comment one line above the code. Errors of that kind become easy to predict, so the loss stays low and truncation keeps them.
This note proposes to measure that failure directly and to fix it with a signal that does not come from the model's own loss. For each code entity a docstring mentions, such as a parameter, a returned variable, or a called function, we check whether the entity actually lies on a data-flow or dependency path to what the function computes. The check uses the program dependency graph, a statement-level graph of which values and branches feed which others. A summary token naming an entity that the graph shows to be dead, absent, or unrelated is removed from the loss whether or not it is surprising. In two sentences: loss-based denoising treats surprise as the signature of noise, but correlated reference errors are not surprising. Program structure offers an independent check on whether a summary's claims touch the code at all.
Core Claim
We make three linked claims. The first is a simulation-to-reality claim. Methods for noisy references are usually validated by injecting independent synthetic corruption into targets, and that kind of simulation overstates how well they will work on real code corpora. The crowdsourcing literature found the analogous gap: Sun et al., who aggregate contributions from workers modeled with Gaussian skill, also observe that real workers share systematic misunderstandings, and majority voting breaks under such errors. Loss truncation is effectively a majority vote held inside the model's training distribution, so it should break the same way.
The second claim is that grounding entity mentions in the data-flow or dependency graph recovers much of the loss truncation misses on correlated noise. It should cost little on clean data, because it only acts on tokens that name code entities and leaves abstractive wording to ordinary truncation.
The third claim concerns contextual noise, meaning misleading or irrelevant text placed in the model's input. A summarizer trained this way should be less likely to echo stale inline comments that appear in its input. The "copying instinct" that Training Dynamics for Text Summarization Models documents early in fine-tuning makes copying such comments cheap under ordinary training.
Research Program
The first component is a family of noise simulators for code references, which turns the sim-to-real question into an experiment. The independent simulator replaces random docstring tokens or entity names, matching how noise is usually injected. The correlated simulators reproduce specific mechanisms. The stale-after-edit simulator applies a behavior-changing edit to a function, such as removing a parameter's effect, changing the return value, or deleting a branch, while leaving its docstring and inline comments untouched. Variable names often survive such edits, so the stale docstring still overlaps lexically with the code. The clone-propagation simulator copies one function's docstring onto near-duplicate functions that differ in behavior. The callee-description simulator replaces the summary with a description of a library function the code calls.
These simulators are useful only if they predict real behavior. We therefore also build a small, human-audited real subset of CodeSearchNet (GraphCodeBERT, CodeXGLUE) Python and Java pairs. Each docstring in it is labeled as consistent or inconsistent with its code, with the inconsistent span marked. The test is whether the ranking of denoising methods under each simulator matches their ranking on the audited real set.
The second component is the training objective. We extract a data-flow graph in the style of GraphCodeBERT, plus statement-level dependency edges as in Boosting Code Summarization by Embedding Code Structures. From these we compute a backward slice: the set of entities that can influence the function's return values or side effects. Target tokens that name code entities form a set $E$, found by matching against the function's identifiers and called names. Each such token gets a grounding score $g_t$, which is 1 if its entity is in the slice and 0 otherwise. Mentions of called library functions get a third value, "unverifiable," because their internals are not visible. The training loss is
$$\mathcal{L} = -\sum_{t=1}^{T} w_t \log P(y_t \mid y_{<t}, X), \qquad w_t = \mathbb{1}[t \notin E \;\text{or}\; g_t \neq 0]\cdot \mathbb{1}[\ell_t < q_{1-c}],$$
Here $X$ is the code, $y_t$ is the $t$-th target token, $\ell_t$ is that token's loss, and $q_{1-c}$ is the running $(1-c)$ loss quantile used for token-level truncation in Training Dynamics for Text Summarization Models. The structural gate removes entity claims the graph contradicts, while the quantile gate continues to handle surprising abstractive tokens. Unverifiable callee mentions fall through to the quantile gate. This choice follows GraphCodeBERT's own caveat that external library data flow cannot be observed.
The third component is a distractor-context evaluation. We prepend stale inline comments produced by the stale-after-edit simulator, or neighboring functions from the same file, to the input. We then measure how often the generated summary repeats the distractor's claims instead of the code's.
Baselines
The naive reference point is plain maximum-likelihood fine-tuning of a pretrained code encoder-decoder on the raw corpus. We use the CodeBERT- or GraphCodeBERT-initialized summarizer family that PBM builds on. Every other method uses the same backbone, data, number of steps, and a matched hyperparameter search of eight configurations. Beating this baseline is necessary but not sufficient.
The strongest competitor on the denoising question is loss truncation itself. We run both the sequence-level version of Improved Natural Language Generation via Loss Truncation and the token-level version of Training Dynamics for Text Summarization Models, tuning the drop fraction $c$ over the same grid we use for our own method. These are the methods our claim says will fail on correlated noise, and on news text they have real successes. On Gigaword, loss truncation reached a human factuality score of 3.79, compared with 3.51 for standard beam search and 3.63 for the human references. On XSum, the factuality variant reduced sentence error rate from 0.67 to 0.59 without a significant ROUGE drop. We found no reported result on Lacuna for loss truncation applied to code summarization, so we will produce that number ourselves.
On the representation question, the strongest competitor is putting structure into the encoder instead of the loss. For this we use PBM from Boosting Code Summarization by Embedding Code Structures, which reported a BLEU improvement of 5.67% on TL-CodeSum and 7.67% on CCSD, and SG-Trans from Code Structure-Guided Transformer for Source Code Summarization, which reported METEOR improvements of 1.4% on Java and 2.0% on Python. Both were trained on references assumed to be clean. Our method must beat them on grounding error under correlated noise. It must also stay within one BLEU point of them on the clean audited subset.
Three ablations isolate the key ingredient. In the first, a lexical filter keeps only the entity mentions whose names appear anywhere in the code, replacing the graph slice with string matching. In the second, the structural gate is used without the quantile gate. In the third, the quantile gate is used without the structural gate, which reproduces token-level truncation. Our method counts as beating the field if, on the audited real set and the correlated simulators, it reduces the grounding error rate relative to the best tuned truncation baseline and the lexical filter, with non-overlapping bootstrap intervals. It must do so without a meaningful clean-data BLEU loss.
Why not just…?
Why not just clean the docstrings with an LLM judge or rewriter before training? This is a serious alternative, and we run it as a baseline using the same audit criteria given to the judge as a prompt. The material gives two reasons for doubt. Less is More found that rewritten text carries the rewriter's own bias about what matters, and that its gains shrank to about 2% on hard problems. A rewriter that reads stale inline comments also faces the same contextual distraction we are trying to measure. If the LLM cleaner matches our method on the audited set at comparable cost, the right conclusion is that a structural gate is unnecessary when a strong judge is available. We would report that outcome.
Why not just check whether each mentioned name appears in the code, with no graph? This is the most dangerous alternative, which is why it is our central ablation. Its weakness is built into the stale-after-edit mechanism: refactors usually leave variable names in place while cutting their effect on the output. A name can therefore be lexically present and still be causally irrelevant. If the lexical filter performs as well as the graph slice on the audited real set, then real stale docstrings rarely take this form. The contribution would then reduce to the correlated-noise diagnosis plus a cheap filter, and we would say so.
Why not just use a structure-aware encoder such as PBM or SG-Trans and keep MLE? A structural encoder helps the model read the code. Maximum likelihood, however, still pays a large loss to reproduce stale claims that the code does not support, and those claims are learnable when they are correlated. The encoder baseline on correlated simulators will show whether better reading alone suppresses them. We expect it does not, because nothing in its objective penalizes copying a stale inline comment.
Why not just truncate more aggressively? Raising $c$ discards more surprising tokens. Correlated errors are not surprising, so a higher drop fraction mainly removes valid abstractive content, which is the known open problem with truncation thresholds. Our sweep over $c$ will show whether any threshold reaches our grounding error without a clean-data quality drop.
Proposed Experiments
We train on CodeSearchNet Python and Java, on the Java and Python summarization corpora from SG-Trans, and on TL-CodeSum, and we test transfer on CCSD and its out-of-domain split. Each training corpus is evaluated in its raw form and under each noise simulator at matched corruption rates. Metrics are BLEU and METEOR against audited-clean references. We also report a grounding error rate, the fraction of generated entity mentions outside the backward slice. A human factual-consistency rating on a sample, scored blind to method, checks that the graph-based metric is not grading itself. In the distractor setting we report an echo rate, the fraction of summaries that reproduce the distractor's stale claim.
The first decisive check is the sim-to-real test. If loss truncation ranks above maximum likelihood under independent noise but not under the correlated simulators or on the audited real set, the core diagnosis holds. We will also report which simulator's method ranking correlates best with the real ranking. If truncation works equally well everywhere, the premise is false and the structural gate has nothing to fix.
The second decisive check is the lexical-versus-graph ablation on stale-after-edit noise and on the real audit set, which separates the value of structure from the value of simply naming entities. The third decisive check is the echo rate under injected stale comments. Here our method should beat both truncation and PBM, because it is the only one whose objective penalizes copying claims the graph does not support.
We will also report the cost of extracting the graph and the share of entity mentions the extractor leaves unverifiable, because Python's dynamic features limit what a static parser can see.
References
- A Model for Aggregating Contributions of Synergistic Crowdsourcing Workflows
- Training Dynamics for Text Summarization Models
- GraphCodeBERT: Pre-training Code Representations with Data Flow
- CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
- Boosting Code Summarization by Embedding Code Structures
- Improved Natural Language Generation via Loss Truncation
- Code Structure Guided Transformer for Source Code Summarization
- Less is More: Summary of Long Instructions is Better for Program Synthesis