Protocode: Prototype-Driven Interpretability for Code Generation in LLMs

Krishna Vamshi Bodla, Haizhao Yang

Introduction

Large Language Models (LLMs) have become central to modern software development, yet they often function as "black boxes," making it difficult to understand the logic behind their generated code. While In-Context Learning (ICL) allows developers to guide these models using demonstration examples, the process of selecting effective "prototypes" remains largely unprincipled. Protocode addresses these challenges by introducing a framework for intelligent prototype selection and syntax-aware interpretability.

Methodology

Protocode employs a two-stage approach to improve both the functional correctness and the transparency of code generation.

Manifold-Based Prototype Sampling

Rather than relying on simple similarity metrics, Protocode uses a neural network to map code embeddings onto a lower-dimensional manifold. By utilizing piecewise-linear manifold learning and Proxy-Anchor metric learning, the system identifies prototypes that are both geometrically representative and semantically distinct across various programming tasks.

AST-Grounded Interpretability

To explain model outputs, Protocode calculates the gradient of the similarity between sampled prototypes and generated tokens. These attribution scores are then mapped onto an Abstract Syntax Tree (AST). This process groups individual tokens into high-level syntax categories, such as "Decisions," "Iterations," and "Exceptions," providing a clear "confidence map" of the model's logic.

AST-Grounded Interpretability Mapping

Evaluation

The framework was evaluated using the Magicoder-OSS-Instruct-75K dataset as a source for prototypes and the MBPP and MBPP+ benchmarks for performance testing. The study included a diverse range of LLMs, including Qwen, Llama, Falcon, and StarCoder.

Results

Experimental results demonstrate that Protocode consistently outperforms standard sampling strategies, such as diversity-based or simple similarity-based selection.

Performance Comparison on MBPP Benchmarks

As shown in the results, Protocode significantly improved the performance of smaller, code-specific models. For instance, the Qwen2.5-coder-0.5B model achieved a pass@10 of 0.122 using Protocode, compared to 0.116 for the base model. Similar trends were observed in the Codellama-7B model.

Discussion and Insights

The AST-based analysis revealed that models exhibit varying "confidence profiles." While most models are highly proficient in handling Data Structures and Scope, they frequently struggle with Exception Handling and complex control flows. This suggests that while models are dependable for structural code, they require more guidance or better prototypes for logic-intensive segments. Furthermore, by utilizing gradients rather than full vocabulary distributions, Protocode achieves this level of interpretability with significantly lower memory overhead than previous state-of-the-art methods.

Conclusion

Protocode provides a robust framework for enhancing LLM-based code generation through principled prototype selection and efficient, syntax-aware explanations. Future research will explore "prototype steering," leveraging these interpretability insights to actively direct models toward specific coding patterns or safety standards.

Summary updated

Abstract

Since the introduction of Large Language Models (LLMs), they have been widely adopted for various tasks such as text summarization, question answering, speech-to-text translation, and more. In recent times, the use of LLMs for code generation has gained significant attention, with tools such as Cursor and Windsurf demonstrating the ability to analyze massive code repositories and recommend relevant changes. Big tech companies have also acknowledged the growing reliance on LLMs for code generation within their codebases. Although these advances significantly improve developer productivity, increasing reliance on automated code generation can proportionally increase the risk of suboptimal solutions and insecure code. Our work focuses on automatically sampling In-Context Learning (ICL) demonstrations which can improve model performance and enhance the interpretability of the generated code. Using AST-based analysis on outputs from the MBPP test set, we identify regions of code most influenced by the chosen demonstrations. In our experiments, we show that high-quality ICL demonstrations not only make outputs easier to interpret but also yield a positive performance improvement on the pass@10 metric. Conversely, poorly chosen ICL demonstrations affected the LLM performance on the pass@10 metric negatively compared to the base model. Overall, our approach highlights the importance of efficient sampling strategies for ICL, which can affect the performance of the model on any given task.
BibTeX
@misc{bodla2025protocode,
  author = {Krishna Vamshi Bodla and Haizhao Yang},
  title = {{Protocode: Prototype-Driven Interpretability for Code Generation in LLMs}},
  year = {2025},
  eprint = {2509.25247},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.25247},
}
Figure