Code Editing from Few Exemplars by Adaptive Multi-Extent Composition

Peizhao Li, Xuchao Zhang, Ziyu Yao, Wei Cheng, Haifeng Chen, Hongfu LiuICLR 2022 · OpenReview

TL;DR

This paper tackles the challenge of automatically editing code snippets based on a few examples, going beyond the limitations of single-example approaches. They introduce a novel method that leverages "multi-extent" similarity matching across abstract syntax trees to compose edit patterns from multiple exemplars, achieving significant accuracy improvements on C# and Python code editing tasks.

What Problem Does This Paper Address?

The task of code editing, which involves modifying code to improve its design, structure, or functionality, is crucial in software engineering. While automating this process is highly desirable for tasks like refactoring or bug fixing, existing deep learning approaches often rely on a single "exemplar" (an original and modified code pair) to learn an editing pattern. This "one-shot" approach can lead to poor generalization and incorrect edits, especially when the exemplar's context is narrow or misleading. The paper addresses this by proposing a method that can effectively learn from multiple exemplars, allowing for more robust and generalized code editing.

Methods

The core of the proposed method lies in its "adaptive multi-extent composition" approach. Here's a breakdown:

  • Abstract Syntax Trees (ASTs): Code snippets are first parsed into ASTs, which represent the hierarchical structure of the code. This allows for more structured analysis than just treating code as plain text.
  • Edit Representations: The system learns "edit representations" from pairs of original and modified code snippets (exemplars). These representations capture the essence of the editing pattern.
  • Multi-Extent Similarity: Instead of just comparing entire ASTs or individual nodes, the method measures similarity at multiple extents. This means it considers matches from individual nodes up to the entire tree structure. A key innovation here is the use of a $\lambda$-softmax function, which allows the model to dynamically adjust the focus between individual nodes and the collective tree representation.
  • Ensemble and Ranking: The similarities computed at different extents are then "ensambled" using a similarity-ranking error estimator. This helps to robustly combine the information from multiple exemplars and extents, prioritizing more relevant editing patterns.
  • Graph2Edit Framework: The proposed method builds upon the existing Graph2Edit framework, which is designed for one-shot code editing. The authors adapt this framework to handle multiple exemplars by incorporating their multi-extent composition mechanism.

Datasets

The paper evaluates their method on two datasets:

  • C#Fixer: This dataset contains editing pairs generated by applying C# fixing tools to existing code. It was previously used in other research.
  • PyFixer: This is a newly collected dataset of Python code editing pairs, generated using the Python-Future tool to ensure compatibility with Python 3.

Key Insights and Ideas

  • The Power of Multiple Exemplars: The paper strongly argues that relying on a single exemplar for code editing is insufficient for achieving robust generalization. Multiple exemplars, even if they exhibit the same general editing pattern, can provide diverse contexts that help the model learn more adaptable rules.
  • Multi-Extent Matching is Crucial: Code editing can be highly context-dependent. Some edits are local (e.g., changing a variable name), while others are global (e.g., refactoring a complex conditional). The multi-extent approach, by considering similarities at various granularities (from single nodes to the entire AST), effectively captures this spectrum of editing needs.
  • Ensembling for Robustness: Simply averaging edit representations from multiple exemplars can be suboptimal. The proposed similarity-ranking error estimator and adaptive ensemble mechanism allow the model to intelligently combine information, leading to more accurate and reliable edits.

Key Results

The proposed method demonstrates significant improvements over baseline approaches:

  • C#Fixer: Achieved up to 8.0% absolute accuracy improvement compared to non-compositional baselines.
  • PyFixer: Achieved up to 10.9% absolute accuracy improvement compared to non-compositional baselines.
  • The paper also shows that performance generally improves as the number of support exemplars increases, indicating the scalability of their approach.

Limitations and Future Work

  • Computational Cost: While the multi-extent approach enhances performance, increasing the number of extents for ensemble can lead to higher computational costs. The authors acknowledge this trade-off.
  • Failure Cases: The paper presents a failure case where complex contexts and concurrent edits from multiple exemplars can mislead the algorithm. This highlights the ongoing challenge of handling highly intricate code transformations.
  • Future Work: The authors suggest that their framework can be developed into a user-friendly tool where users can collect exemplars to guide the editing process, offering flexibility and personalization. They also mention releasing their collected PyFixer dataset and model implementation.

Summary updated

Abstract

This paper considers the computer source code editing with few exemplars. The editing exemplar, containing the original and modified support code snippets, showcases a certain editorial style and implies the edit intention for a query code snippet. To achieve this, we propose a machine learning approach to adapt the editorial style derived from few exemplars to a query code snippet. Our learning approach combines edit representations extracted from editing exemplars and compositionally generalizes them to the query code snippet editing via multi-extent similarities ensemble. Specifically, we parse the code snippets using language-specific grammar into abstract syntax trees. We apply the similarities measurement in multiple extents from individual nodes to collective tree representations, and ensemble them through a similarity-ranking error estimator. We evaluate the proposed method on two datasets in C\# and Python languages and respectively show 8.0\% and 10.9\% absolute accuracy improvements compared to baselines.
BibTeX
@inproceedings{li2022code,
  author = {Peizhao Li and Xuchao Zhang and Ziyu Yao and Wei Cheng and Haifeng Chen and Hongfu Liu},
  title = {{Code Editing from Few Exemplars by Adaptive Multi-Extent Composition}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=i7O3VGpb7qZ},
}
Figure