xASTNN: Improved Code Representations for Industrial Practice

Zhiwei Xu, Min Zhou, Xibin Zhao, Yang Chen, Xi Cheng, Hongyu Zhang

Abstract

While deep learning has significantly advanced source code representation, existing models often fail to meet the rigorous demands of industrial production. This paper presents xASTNN (eXtreme Abstract Syntax Tree-based Neural Network), a high-performance architecture designed for efficiency, scalability, and language-agnostic deployment. By utilizing a gated recursive architecture and a novel dynamic batching algorithm, xASTNN achieves state-of-the-art accuracy on code classification and clone detection tasks while delivering a 10x speedup over previous models.

1. Introduction

The transition of code embedding models from research environments to industrial-scale applications is hindered by significant bottlenecks. Current state-of-the-art (SOTA) models typically suffer from three primary "deal-breakers":

  1. Efficiency Bottlenecks: High computational costs during training and deployment.
  2. Deployment Difficulties: Requirements for complex, language-specific compilers or elaborate graph extractions.
  3. Gradient Instability: Difficulties in processing long code segments or deep syntax trees due to vanishing or exploding gradients.

xASTNN addresses these challenges by providing a lightweight, stable, and highly parallelizable framework for representing source code.

2. The xASTNN Framework

The xASTNN pipeline is structured into two primary phases: statement subtree extraction and a dual-gated encoder architecture.

2.1 Statement Subtree Extraction

Rather than treating code as a linear sequence of tokens or a monolithic graph, xASTNN decomposes source code into an Abstract Syntax Tree (AST) and subsequently partitions it into a sequence of statement subtrees. This approach leverages "code naturalness," capturing local syntax within statements and global flow across the sequence of statements. This granularity strikes a balance between structural richness and computational simplicity.

2.2 Encoder Architecture

The model employs two specialized units to process the extracted subtrees:

  • Gated Recursive Unit (GRvU): Designed to encode the syntax of individual statements. The gating mechanism allows the model to selectively retain or discard information, effectively mitigating gradient instability in deep syntax trees. Furthermore, its position-insensitive nature reduces overall computational complexity.
  • Gated Recurrent Unit (GRtU): A standard bidirectional GRU is utilized to capture the sequential dependencies between the encoded statements.

2.3 Dynamic Batching Optimization

The core of xASTNN's performance lies in its dynamic batching algorithm. Traditional tree-based models are notoriously difficult to parallelize due to varying tree structures. xASTNN flattens subtrees and processes nodes of the same depth across an entire batch simultaneously. This depth-wise parallelism ensures maximum GPU utilization and significantly reduces the sequential overhead inherent in recursive structures.

3. Experimental Evaluation

The performance of xASTNN was evaluated across three benchmark datasets:

  • POJ: 52,000 C programs used for code classification (identifying problem solutions).
  • BigCloneBench: A large-scale Java dataset for code clone detection.
  • OJClone: A C-language dataset for clone detection derived from the POJ entries.

4. Results and Discussion

4.1 Performance and Accuracy

xASTNN demonstrated superior performance compared to established baselines, including CodeBERT and GraphCodeBERT. Notably, it achieved an accuracy of 0.985 on the POJ classification task.

4.2 Efficiency and Industrial Readiness

In comparative testing, xASTNN was 9.3x faster than the original ASTNN, requiring only 0.23 milliseconds to represent a single code segment. Beyond raw speed, the model exhibited significantly lower GPU memory consumption. While previous models encountered memory overflows at batch sizes exceeding 128, xASTNN maintained stability at larger scales, making it a cost-effective solution for cloud-based production environments.

5. Conclusion and Future Work

xASTNN represents a significant advancement in making structural code representations viable for industry. Future research will focus on addressing the challenges of unbalanced trees within the dynamic batching algorithm and conducting extensive robustness testing to evaluate performance under "noisy" or abnormal code inputs. Additionally, the development of universal parsers could further enhance the model's language-agnostic capabilities.

Summary updated

Abstract

The application of deep learning techniques in software engineering becomes increasingly popular. One key problem is developing high-quality and easy-to-use source code representations for code-related tasks. The research community has acquired impressive results in recent years. However, due to the deployment difficulties and performance bottlenecks, seldom these approaches are applied to the industry. In this paper, we present xASTNN, an eXtreme Abstract Syntax Tree (AST)-based Neural Network for source code representation, aiming to push this technique to industrial practice. The proposed xASTNN has three advantages. First, xASTNN is completely based on widely-used ASTs and does not require complicated data pre-processing, making it applicable to various programming languages and practical scenarios. Second, three closely-related designs are proposed to guarantee the effectiveness of xASTNN, including statement subtree sequence for code naturalness, gated recursive unit for syntactical information, and gated recurrent unit for sequential information. Third, a dynamic batching algorithm is introduced to significantly reduce the time complexity of xASTNN. Two code comprehension downstream tasks, code classification and code clone detection, are adopted for evaluation. The results demonstrate that our xASTNN can improve the state-of-the-art while being faster than the baselines.
BibTeX
@inproceedings{xu2023xastnn,
  author = {Zhiwei Xu and Min Zhou and Xibin Zhao and Yang Chen and Xi Cheng and Hongyu Zhang},
  title = {{xASTNN: Improved Code Representations for Industrial Practice}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2023},
  doi = {10.1145/3611643.3613869},
}
Figure