xASTNN: Improved Code Representations for Industrial Practice

Zhiwei Xu, Min Zhou, Xibin Zhao, Yang Chen, Xi Cheng, Hongyu Zhang

Abstract

While deep learning has significantly advanced source code representation, existing models often fail to meet the rigorous demands of industrial production. This paper presents xASTNN (eXtreme Abstract Syntax Tree-based Neural Network), a high-performance architecture designed for efficiency, scalability, and language-agnostic deployment. By utilizing a gated recursive architecture and a novel dynamic batching algorithm, xASTNN achieves state-of-the-art accuracy on code classification and clone detection tasks while delivering a 10x speedup over previous models.

1. Introduction

The transition of code embedding models from research environments to industrial-scale applications is hindered by significant bottlenecks. Current state-of-the-art (SOTA) models typically suffer from three primary "deal-breakers":

  1. Efficiency Bottlenecks: High computational costs during training and deployment.
  2. Deployment Difficulties: Requirements for complex, language-specific compilers or elaborate graph extractions.
  3. Gradient Instability: Difficulties in processing long code segments or deep syntax trees due to vanishing or exploding gradients.

xASTNN addresses these challenges by providing a lightweight, stable, and highly parallelizable framework for representing source code.

2. The xASTNN Framework

The xASTNN pipeline is structured into two primary phases: statement subtree extraction and a dual-gated encoder architecture.

2.1 Statement Subtree Extraction

Rather than treating code as a linear sequence of tokens or a monolithic graph, xASTNN decomposes source code into an Abstract Syntax Tree (AST) and subsequently partitions it into a sequence of statement subtrees. This approach leverages "code naturalness," capturing local syntax within statements and global flow across the sequence of statements. This granularity strikes a balance between structural richness and computational simplicity.

2.2 Encoder Architecture

The model employs two specialized units to process the extracted subtrees:

  • Gated Recursive Unit (GRvU): Designed to encode the syntax of individual statements. The gating mechanism allows the model to selectively retain or discard information, effectively mitigating gradient instability in deep syntax trees. Furthermore, its position-insensitive nature reduces overall computational complexity.
  • Gated Recurrent Unit (GRtU): A standard bidirectional GRU is utilized to capture the sequential dependencies between the encoded statements.

2.3 Dynamic Batching Optimization

The core of xASTNN's performance lies in its dynamic batching algorithm. Traditional tree-based models are notoriously difficult to parallelize due to varying tree structures. xASTNN flattens subtrees and processes nodes of the same depth across an entire batch simultaneously. This depth-wise parallelism ensures maximum GPU utilization and significantly reduces the sequential overhead inherent in recursive structures.

3. Experimental Evaluation

The performance of xASTNN was evaluated across three benchmark datasets:

  • POJ: 52,000 C programs used for code classification (identifying problem solutions).
  • BigCloneBench: A large-scale Java dataset for code clone detection.
  • OJClone: A C-language dataset for clone detection derived from the POJ entries.

4. Results and Discussion

4.1 Performance and Accuracy

xASTNN demonstrated superior performance compared to established baselines, including CodeBERT and GraphCodeBERT. Notably, it achieved an accuracy of 0.985 on the POJ classification task.

4.2 Efficiency and Industrial Readiness

In comparative testing, xASTNN was 9.3x faster than the original ASTNN, requiring only 0.23 milliseconds to represent a single code segment. Beyond raw speed, the model exhibited significantly lower GPU memory consumption. While previous models encountered memory overflows at batch sizes exceeding 128, xASTNN maintained stability at larger scales, making it a cost-effective solution for cloud-based production environments.

5. Conclusion and Future Work

xASTNN represents a significant advancement in making structural code representations viable for industry. Future research will focus on addressing the challenges of unbalanced trees within the dynamic batching algorithm and conducting extensive robustness testing to evaluate performance under "noisy" or abnormal code inputs. Additionally, the development of universal parsers could further enhance the model's language-agnostic capabilities.

Summary updated

BibTeX
@inproceedings{xu2023xastnn,
  author = {Zhiwei Xu and Min Zhou and Xibin Zhao and Yang Chen and Xi Cheng and Hongyu Zhang},
  title = {{xASTNN: Improved Code Representations for Industrial Practice}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2023},
  doi = {10.1145/3611643.3613869},
}
Figure