Pointing Near the Source: Domain-Embedding Neighborhoods for Copying in Code Summarization

Models that name a method or describe it in a sentence often copy words straight from the code. A pointer-generator is a decoder that, at each step, either produces a word from its fixed vocabulary or copies a word from the input. It is the standard tool for this, and it works best when the target word appears verbatim in the source. Developers, however, often write a close relative of a source word instead of the word itself: the body says increase and the name says increment, the body calls remove and the comment says "deletes", the body reads a size field and the name is getCount. A copy mechanism that only reproduces exact strings gets no credit for having found the right source location. A generator with a fixed vocabulary cannot produce the relative if it is rare. The result is a gap between "the model knows where the answer is" and "the model can say it."

We propose to close that gap with a pointer whose output is a small neighborhood rather than a single string. The model points to a leaf of the program's syntax tree, then chooses either that leaf's subword or one of its nearest neighbors in a word-embedding space trained on developer text such as Stack Overflow. The domain embedding does not serve as a better initialization here. It decides which words count as "near" a copied token, and it can supply neighbors that lie outside the decoder's own vocabulary. The structural code representation fixes where the pointer can land. The domain embedding fixes what it can emit once it lands.

Core Claim

The closest existing system is the hierarchical copy model of Pointing to Subwords for Generating Function Names in Source Code. It extends code2seq, which represents a method as a bag of paths through its abstract syntax tree (AST, the parse tree of the code), each path running between two leaf tokens. The hierarchical copy model first attends over these paths, then over the leaf subwords within the chosen path, and copies the selected subword exactly. It raised F1** on Java-small from 40.69 to 45.55. It lifted accuracy on very rare words from near 0% to 10.69%. The authors also point out that exact-match scoring punishes valid synonyms.

Our claim is that a large share of the remaining errors are "near-copies." In these cases the model attends to the correct leaf, but the reference uses a related form of that leaf's word: a synonym, a more specific API term, or an inflection. We claim a copy distribution spread over a domain-embedding neighborhood recovers these cases under the standard exact-match metric. We also claim the neighborhood must come from software-domain text: neighborhoods from general-purpose embeddings or from string similarity should help less.

Concretely, let $x_i$ be the subword at leaf $i$, let $a_i$ be the hierarchical (path-then-leaf) attention weight on it, and let $N_k(x_i)$ be $x_i$ together with its $k$ nearest neighbors in a frozen domain embedding space. The output distribution is

$$P(w) = p_{gen}\,P_{vocab}(w) + (1-p_{gen})\sum_i a_i\, q(w \mid x_i, h_t, r_i),$$

Here $p_{gen}$ is the usual learned switch between generating and copying, $h_t$ is the decoder state, and $r_i$ is the encoding of the AST path containing leaf $i$. The term $q$ is a softmax over $N_k(x_i)$, scored by a bilinear match between $h_t$ and the frozen neighbor embeddings, with a learned bonus for the exact token. Setting $k=0$ recovers the hierarchical copy model exactly, so every gain can be attributed to the neighborhood.

Conditioning $q$ on $r_i$ lets the syntax tree shape the substitution. The same leaf size should map to count when it sits in a field-access path feeding a return statement, and stay size when it is an argument to a constructor.

Research Program

The first step is a measurement that can stop the project before any model is built. On Java-small and on the method–comment pairs of the CoSTop corpus (A Topic Guided Pointer-Generator Model), we compute the "near-copy rate." This is the fraction of target subwords that do not occur in the method body but do have a body subword whose domain-embedding neighborhood ($k \le 10$) contains them. We report it separately for neighbors that are pure inflections or edit-distance variants and for genuine lexical substitutes. For calibration, that paper found that about 33% of comment words appear verbatim in the code, and Fujita et al. found that the most frequent body subword appears in the name over 31% of the time. If near-copies add only a couple of points beyond exact copies, or if nearly all of them are inflections, the idea reduces to stemming and we stop.

The second step builds the domain space. We train skip-gram embeddings over subword-split Stack Overflow text, following the choice of Req2Lib. That work showed such embeddings link intent words to technical terms, for example "logging" to log4j. We exclude posts that quote code from evaluation projects, to avoid leakage. The space stays frozen, so its vocabulary can be larger than the decoder's and supply neighbors the generator cannot produce.

The third step trains the model on Java-small, Java-large, and CodeNN, on top of the code2seq encoder with Fujita et al.'s two-level attention.

The fourth step is analysis. For each correct prediction we record whether it came from the generator, from an exact copy, or from a neighbor copy. We then examine what the AST conditioning learned by checking which path types license substitution.

This design takes a specific position on two findings in the file. A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization found that content selection transfers across domains while vocabulary does not. In our model, content selection lives in the pointer attention and domain vocabulary lives in the neighborhood, so the two are separated by construction. AST-T5 reports that strongly structure-biased models generate natural language worse. Our design keeps structure where it helps, in locating the evidence, and hands word choice to a text-trained space. The comment-generation experiments test whether that split avoids the degradation AST-T5 describes.

Baselines

The naive reference point is a frequency rule. For naming, it emits the most frequent body subwords in source order, the regularity Fujita et al. exploit with their most-frequent-subword placeholder. For comments, it is a TF-IDF nearest-neighbor retrieval that returns the comment of the most similar training method. The second is the kind of baseline that A Comprehensive Study on the Use of Word Embedding Models in Software Engineering Domain found missing from 84% of the studies it reviewed. A neural model that cannot clearly beat these has no claim.

The strongest existing method on our main benchmark is the hierarchical copy model of Pointing to Subwords, which reports 45.55 F1 on Java-small against 40.69 for code2seq. We reimplement it and use it as the $k=0$ configuration. Our model therefore shares its encoder, attention, data split, vocabulary, and tuning budget, and differs only in the neighborhood term. Beating it means a gain in F1 over our own reimplementation, with the reimplementation reproduced to within about one point of 45.55. The gain must also be statistically reliable across three seeds, and most of it must come from predictions that the attribution analysis traces to neighbor copies.

On Java-large we compare against code2seq at its reported F1 of 59.19. Fujita et al. report no exact Java-large number in the file. For CodeNN captioning we compare against code2seq's reported BLEU of 23.04. For the CoSTop comment task, the file contains no scalar result for CoSTop/ToPNN, only curves. We therefore retrain a pointer-generator over flat tokens as that paper's comparison point and report our own numbers for it.

Four ablations isolate the key ingredient. Each keeps everything fixed except where the neighborhood comes from. The first builds neighborhoods from general-domain embeddings trained on Wikipedia-scale text with the same recipe. The second uses string-edit-distance neighbors only. The third uses randomly drawn vocabulary words, which controls for the extra output capacity alone. The fourth initializes the generator's embeddings from the Stack Overflow space with no neighborhood copying at all. We also remove the AST path term $r_i$ from $q$, to test whether structure matters for substitution or only for locating the leaf.

Why not just…?

Why not just initialize the decoder's embeddings with Stack Overflow vectors? This is the obvious way to combine the three directions, and it is a real threat. If the generator already produces increment whenever its state resembles increase, the neighborhood adds nothing. Two arguments say it may fall short. First, a generator can only emit words in its own vocabulary, and Fujita et al. show that rare words are where generators score near 0%. Second, initialization effects tend to wash out under training on hundreds of thousands of examples. The "domain-initialized generator, no neighborhood" ablation settles this directly. If it matches our model on F1**, we report that the simpler method suffices.

Why not just stem or lemmatize the copied token? If near-copies are mostly add/adds/adder, string similarity captures them cheaply. The near-copy measurement separates inflectional from lexical neighbors before training. The edit-distance ablation then tests whether domain neighbors add anything beyond morphology. We expect them to matter for API-level substitutes such as size→count or remove→delete, but this is an empirical question, and a null result would narrow the claim to morphology.

Why not just use a large pretrained code model with subword tokenization? Subword decoders can in principle spell any word. However, the code2seq baseline in Fujita et al. already uses subwords and still fails on rare terms. AST-T5 reports strong code-generation results (14.0 Pass@1 on HumanEval) but weaker natural-language generation, and the file contains no reported result for it on Java-small naming. We include a fine-tuned AST-T5-sized encoder-decoder on Java-small and CoSTop comments as an external reference. We will state plainly if it dominates, in which case the contribution becomes the neighborhood head added on top of it.

Why not just score with a semantic metric instead of changing the model? Rescoring with embedding similarity would reward synonyms without any modeling change. It would also be circular if it used the same embeddings the model uses. Our primary metric is therefore unchanged exact-match F1/F1**. We report a synonym-tolerant score only from a separately trained embedding space, and we add a small human judgment of whether names are acceptable.

Proposed Experiments

The decisive checks run in order. The near-copy rate on Java-small and CoSTop must show that lexical (non-inflectional) near-copies account for a meaningful share of the subwords that exact copying misses. Our model must then beat the reimplemented hierarchical copy baseline on Java-small F1**, and must not lose to code2seq's reported 59.19 F1 on Java-large or 23.04 BLEU on CodeNN.

The Stack Overflow neighborhood must beat the general-embedding, edit-distance, and random-neighborhood versions. If the general-embedding version matches it, the "domain-specific" part of the claim fails even if the mechanism works.

On rare targets, we report accuracy in the same frequency bucket Fujita et al. used, where their model reached 10.69%. Gains should concentrate on words that are absent from the decoder vocabulary but present in the domain space, the regime the mechanism is built for.

Finally, on CoSTop comment generation we check, using BLEU, ROUGE-L, and METEOR, whether the gain carries over from method names to full sentences. If it does not, the result is useful too: neighborhood copying would then be suited to short, identifier-like outputs and not to prose.