Towards Generating Summaries for Lexically Confusing Code through Code Erosion

Fan Yan, Ming LiIJCAI 2021 · OpenReview

Abstract

Code summarization aims to summarize code functionality as high-level natural language descriptions to assist in code comprehension. Recent approaches in this field mainly focus on generating summaries for code with precise identifier names, in which meaningful words can be found indicating code functionality. When faced with lexically confusing code, current approaches are likely to fail since the correlation between code lexical tokens and summaries is scarce. To tackle this problem, we propose a novel summarization framework named VECOS. VECOS introduces an erosion mechanism to conquer the model's reliance on precisely defined lexical information. To facilitate learning the eroded code's functionality, we force the representation of the eroded code to align with the representation of its original counterpart via variational inference. Experimental results show that our approach outperforms the state-of-the-art approaches to generate coherent and reliable summaries for various lexically confusing code.

Introduction

Modern software development relies heavily on automated code summarization to help developers understand complex codebases. However, these models often depend on "lexical precision"—the presence of meaningful variable and function names like calculate_interest_rate. In practice, code is frequently "lexically confusing" due to obfuscation for intellectual property protection or poor naming conventions. When these semantic clues are removed or misleading, standard models that correlate tokens to summaries tend to fail. This paper introduces VECOS (Variational Eroded COde Summarization), a framework designed to generate summaries based on the structural logic of code rather than its identifiers.

Methodology: The VECOS Framework

The VECOS framework addresses lexical confusion through two primary innovations: a code erosion mechanism and a variational alignment strategy.

The Erosion Mechanism ($\tau$)

To prevent the model from becoming "lazy" by relying on identifier names, VECOS employs a rule-based preprocessing step called Code Erosion. This process replaces all user-defined identifiers—including function names, parameters, and local variables—with a standardized set of generic symbols (e.g., func0, para0, var0). This strips away the lexical semantics while preserving the underlying functional logic and Abstract Syntax Tree (AST) structure.

The Code Erosion process transforms meaningful identifiers into generic symbols to preserve structural logic while removing lexical clues.

Variational Alignment

Stripping identifiers makes the summarization task significantly harder. To compensate, VECOS uses a variational inference approach to align the representations of eroded code with their original, lexically precise counterparts. An auxiliary model pre-trained on original code learns a latent representation ($z$) of functionality. The summarization model, which only sees the eroded code, is then trained to produce a representation that aligns with this latent space. This allows the model to internally map generic structures back to the functional concepts they represent.

Experimental Evaluation

The researchers evaluated VECOS using a Java dataset of 274,302 code-summary pairs. To simulate real-world lexical confusion, they tested the model against two types of noise at various ratios (0% to 100%):

  1. Symbolic Replacement: Simulating obfuscation by replacing identifiers with ordered symbols.
  2. Synonym Replacement: Simulating poor naming by replacing sub-tokens with synonyms using GloVe embeddings.

Key Results

While baseline models like the Transformer and AttnGRU experienced significant performance drops as the ratio of lexical confusion increased, VECOS demonstrated remarkable stability.

Comparison of BLEU scores across different models and replacement ratios, highlighting the stability of VECOS compared to standard approaches.

As shown in the performance data, VECOS maintained a consistent BLEU score of 26.28 even at 100% obfuscation. In contrast, standard models saw their scores plummet from the mid-20s to below 10. VECOS also significantly outperformed AST-AttendGRU, a model specifically designed for code without documentation, by over 10 BLEU points. Ablation studies confirmed that the variational alignment was critical; removing it resulted in a 10% performance decrease.

Conclusion

VECOS demonstrates that structural semantics are sufficient for generating high-quality code summaries, even when identifier names are entirely absent. By forcing the model to focus on logic rather than labels, VECOS achieves a level of robustness previously unseen in the field. This method of learning "functional semantics" in a latent space could eventually be applied to other tasks, such as code clone detection or vulnerability discovery, where obfuscation is a common hurdle.

Summary updated

Abstract

Code summarization aims to summarize code functionality as high-level nature language descriptions to assist in code comprehension. Recent approaches in this field mainly focus on generating summaries for code with precise identifier names, in which meaningful words can be found indicating code functionality. When faced with lexically confusing code, current approaches are likely to fail since the correlation between code lexical tokens and summaries is scarce. To tackle this problem, we propose a novel summarization framework named VECOS. VECOS introduces an erosion mechanism to conquer the model's reliance on precisely defined lexical information. To facilitate learning the eroded code's functionality, we force the representation of the eroded code to align with the representation of its original counterpart via variational inference. Experimental results show that our approach outperforms the state-of-the-art approaches to generate coherent and reliable summaries for various lexically confusing code.
BibTeX
@inproceedings{yan2021towards,
  author = {Fan Yan and Ming Li},
  title = {{Towards Generating Summaries for Lexically Confusing Code through Code Erosion}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=1wCxNnhRYW},
}
Figure