Cross-Lingual Adversarial Domain Adaptation for Novice Programming

Ye Mao, Farzaneh Khoshnevisan, Thomas W. Price, Tiffany Barnes, Min ChiAAAI 2022 · DOI

Introduction

Student Modeling in Programming (SMP) represents a significant challenge in adaptive learning: predicting when a student is struggling to provide timely interventions. While deep learning models like the Abstract Syntax Tree Neural Network (ASTNN) excel at analyzing code structure, they typically require massive datasets. In real-world classroom settings, data is often scarce, limited to a few dozen students. This "data scarcity" is exacerbated by the open-ended nature of programming, where infinite paths can lead to a correct solution.

To address this, researchers developed CrossLing, a framework that leverages knowledge from larger datasets in different programming languages (e.g., Java) to improve performance on smaller, niche datasets (e.g., block-based coding). By separating universal programming logic from language-specific syntax through adversarial learning, CrossLing significantly outperforms existing methods in early success prediction.

Methodology: Adversarial Domain Adaptation

The core of CrossLing is its ability to distinguish between "global" features (universal logic like loops and variables) and "local" features (domain-specific syntax).

The CrossLing architecture utilizes adversarial training with a Gradient Reversal Layer (GRL) to extract domain-invariant global features while maintaining domain-specific local features.

As shown in Figure 1, the framework employs several key components:

  1. Adversarial Training: A Gradient Reversal Layer (GRL) and a discriminator are used to ensure the global encoder learns representations that are domain-invariant. The discriminator tries to identify the source dataset, while the encoder is trained to "confuse" it.
  2. Difference Loss: This loss function ensures that local and global latent spaces remain orthogonal, preventing the mixing of shared logic and specific syntax.
  3. Temporal Integration: For time-series prediction, the authors integrated Time-aware LSTMs (T-LSTM). This allows the model to capture the rhythm of student edits, which is a strong indicator of proficiency.

Experimental Setup and Datasets

The study evaluated CrossLing using two distinct domains:

  • iSnap ($D_1$): A small dataset from a block-based environment (Snap!) containing fine-grained trace logs of student actions.
  • CodeWorkout ($D_2$): A larger dataset of Java submissions, containing only final code states.

The primary objective was to use the larger Java dataset to enhance predictive accuracy for the smaller Snap! dataset.

Key Results and Performance

CrossLing was tested on two primary tasks: Program Classification (correctness) and Early Success Prediction (completion within a set timeframe).

Critical Difference (CD) diagram showing the mean rank of various models. TL-CrossLing significantly outperforms baselines and standard ASTNN models.

The results demonstrate the framework's superiority:

  • Program Classification: CrossLing achieved an AUC of 0.827, surpassing the standard ASTNN (0.808) and traditional ML baselines.
  • Early Success Prediction: In the critical early stages (e.g., at the 2-minute mark), the time-aware TL-CrossLing model showed a 9% improvement in AUC over the best deep learning baseline and a 15% improvement over expert-feature models.
  • Consistency: As illustrated in Figure 6, CrossLing variants consistently ranked as the top-performing models across various observation windows (from 2 to 10 minutes).

Insights and Future Directions

The success of CrossLing underscores that programming logic is largely universal and transferable across languages. Visualization via t-SNE confirmed that CrossLing creates more distinct clusters of successful versus unsuccessful students compared to non-adversarial models, which often muddle these groups in their latent space.

Future work will explore expanding this to "Multi-Domain" adaptation, incorporating multiple languages like Python and C++ simultaneously. Additionally, the researchers aim to transition from prediction to active intervention, using these models to trigger real-time automated hints or alerts for educators in the classroom.

Summary updated

BibTeX
@inproceedings{mao2022cross,
  author = {Ye Mao and Farzaneh Khoshnevisan and Thomas W. Price and Tiffany Barnes and Min Chi},
  title = {{Cross-Lingual Adversarial Domain Adaptation for Novice Programming}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i7.20735},
}
Figure