Abstract
Large Language Models (LLMs) have been demonstrated effective for code generation. Due to the complexity and opacity of LLMs, little is known about how these models generate code. To deepen our understanding, we investigate whether LLMs attend to the same parts of a natural language description as human programmers during code generation. An analysis of five LLMs on a popular benchmark, HumanEval, revealed a consistent misalignment between LLMs' and programmers' attention. Furthermore, we found that there is no correlation between the code generation accuracy of LLMs and their alignment with human programmers. Through a quantitative experiment and a user study, we confirmed that, among twelve different attention computation methods, attention computed by the perturbation-based method is most aligned with human attention and is constantly favored by human programmers. Our findings highlight the need for human-aligned LLMs for better interpretability and programmer trust.
Introduction: The Black Box of AI Coding
While Large Language Models (LLMs) like GPT-4 have become remarkably proficient at generating code, they often function as "black boxes." We see the code they produce, but the internal logic remains opaque. This creates a significant trust gap: if a model generates a correct solution, did it truly understand the requirements, or did it simply find a statistical shortcut? Conversely, when it fails, is it because it ignored a crucial constraint?
This study investigates whether LLMs pay attention to the same "salient" parts of a natural language task description—such as data types, operators, and corner cases—as human programmers. By quantifying this "attention alignment," the researchers aim to improve model interpretability and identify the root causes of specific coding errors.
Methodology: Quantifying "Attention"
The researchers conducted a multi-stage empirical study utilizing two industry-standard benchmarks: HumanEval (164 tasks) and MBPP (974 tasks).
Establishing a Human Baseline
To create a ground truth, the authors introduced the Programmer Attention Dataset. Experienced programmers were tasked with manually labeling essential keywords in 1,138 programming prompts, identifying the tokens they deemed critical for solving the task.
Extracting Model Attention
The study tested 12 different methods to calculate what the models "cared about," categorized into three main approaches:
- Self-attention-based: Analyzing the internal weights of the Transformer layers.
- Gradient-based: Using saliency maps to identify which input tokens most affect the output when slightly modified.
- Perturbation-based: Systematically masking or deleting words in the prompt (using tools like SHAP or BERT-masking) to observe changes in the generated code.
Alignment was measured using robust statistical metrics, including San Martino’s token overlap and Krippendorff’s alpha, to compare the human "gold standard" against the model's attention maps.
Key Results: The Alignment Gap
The study revealed a consistent and significant misalignment between LLM and human attention. Across all models and methods, Krippendorff’s alpha remained below 0.3, indicating very weak agreement.
The Reasoning Gap and Performance
Surprisingly, as models become more capable, they do not necessarily become more human-aligned. GPT-4 showed the lowest overlap with humans among all models tested, despite having the highest functional correctness (67% Pass@1). This suggests that larger models may develop reasoning strategies that are increasingly divergent from human logic.
Attention as an Error Predictor
The researchers found that approximately 27% of code generation errors could be explained by specific attention failures. They identified five specific failure patterns:
- Missing critical conditions: Ignoring constraints like "if the list is empty."
- Missing descriptive words: Ignoring modifiers like "ascending order."
- Missing operation descriptions: Overlooking verbs like "sort" or "count."
- Missing data types: Failing to recognize the required input/output types.
- Incorrect mapping: Attending to the correct word but applying the wrong logic in the code.
Discussion and Future Directions
The study concluded that perturbation-based methods (like BERT-masking) produced attention scores most aligned with human intuition. In contrast, internal "self-attention" weights—frequently used in research to explain models—were the least aligned with what humans found important.
While the study was limited to Python and a sample of 22 participants for the user study, the implications are significant. The authors suggest that the Programmer Attention Dataset could be used to "fine-tune" LLMs. By penalizing models when their attention deviates from human focus, researchers may create AI that is not only more accurate but also more interpretable and trustworthy for human developers.
Summary updated