A Study of Variable-Role-based Feature Enrichment in Neural Models of Code

Aftab Hussain, Md. Rafiqul Islam Rabin, Bowen Xu, David Lo, Mohammad Amin Alipour

Abstract

Although deep neural models substantially reduce the overhead of feature engineering, the features readily available in the inputs might significantly impact training cost and the performance of the models. In this paper, we explore the impact of an unsupervised feature enrichment approach based on variable roles on the performance of neural models of code. The notion of variable roles has been found to help students' abilities in programming. We investigate if this notion would improve the performance of neural models of code. In particular, we enrich a source code dataset by adding the role of individual variables and conduct a study on the impact of variable role enrichment in training the Code2Seq model.

Introduction

Training deep neural models for code intelligence is resource-intensive. To improve efficiency and performance, researchers often employ feature enrichment—adding explicit semantic hints to input data. This study investigates variable roles, a pedagogical concept used in computer science education to help students understand program flow. Roles like "steppers" (loop indices) and "walkers" (structure traversers) provide mental models for humans. The researchers hypothesized that providing these explicit labels to AI models could improve accuracy or robustness.

Methods

The researchers implemented a two-stage pipeline for feature enrichment:

  1. Role Detection: A static analyzer using javalang identified two primary roles:
  2. Steppers: Numeric variables updated arithmetically within for loops (e.g., i++).
  3. Walkers: Variables used in enhanced for loops to traverse data structures.
  4. Role Augmentation: Variable names were modified by prefixing them with their detected role (e.g., a variable named count becomes stepper_count).
The role detection and augmentation pipeline.

The study utilized the JAVA-LARGE (JL) dataset and its augmented counterpart, JAVA-LARGE-ROLES (JLR). The Code2Seq model, which represents code using paths in an Abstract Syntax Tree (AST), was trained on these datasets to predict method names.

Results

The primary finding of the study was a "negative result": explicit role enrichment had a negligible impact on model performance. As shown in the results table, the F1-scores for the original (O) and role-augmented (R) models were nearly identical across all test configurations.

Comparison of Precision, Recall, and F1-Score for original (O) and role-augmented (R) models across various test sets.

On the standard JL test set, both models achieved an F1-score of approximately 0.584. Even when filtering the test set to only include methods containing augmented variables (the "Filtered" sets), the performance delta remained marginal (often less than 0.01). To test robustness, the researchers applied "noise induction" by replacing variable names with generic placeholders (e.g., var1). Even in these transformed scenarios, the role-augmented model provided no significant advantage.

Discussion and Insights

Several factors may explain why pedagogical roles did not translate to neural performance gains:

  • Structural Redundancy: Models like Code2Seq likely capture variable roles implicitly through AST structure and context. Explicit labels may offer no novel information to the network.
  • Naming Conventions: A staggering 76% of detected "steppers" were named i. The model likely already associates such common tokens with loop-indexing behavior.
  • Low Coverage: Only about 8% of variables in the 14-million-method dataset were identified as steppers or walkers. A more diverse set of roles might be necessary to see a measurable effect.

Conclusion

While variable roles are effective for human learning, they do not currently provide a performance boost for AST-based models like Code2Seq. The study suggests that neural models are already adept at extracting these patterns from raw structural data. Future work should explore more complex data-flow analysis for role detection and evaluate these enrichments on different architectures, such as Large Language Models (LLMs).

Summary updated

Abstract

Although deep neural models substantially reduce the overhead of feature engineering, the features readily available in the inputs might significantly impact training cost and the performance of the models. In this paper, we explore the impact of an unsuperivsed feature enrichment approach based on variable roles on the performance of neural models of code. The notion of variable roles (as introduced in the works of Sajaniemi et al. [Refs. 1,2]) has been found to help students' abilities in programming. In this paper, we investigate if this notion would improve the performance of neural models of code. To the best of our knowledge, this is the first work to investigate how Sajaniemi et al.'s concept of variable roles can affect neural models of code. In particular, we enrich a source code dataset by adding the role of individual variables in the dataset programs, and thereby conduct a study on the impact of variable role enrichment in training the Code2Seq model. In addition, we shed light on some challenges and opportunities in feature enrichment for neural code intelligence models.
BibTeX
@inproceedings{hussain2023study,
  author = {Aftab Hussain and Md. Rafiqul Islam Rabin and Bowen Xu and David Lo and Mohammad Amin Alipour},
  title = {{A Study of Variable-Role-based Feature Enrichment in Neural Models of Code}},
  booktitle = {InteNSE@ICSE},
  year = {2023},
  doi = {10.1109/intense59150.2023.00007},
}
Figure