Introduction
Large Language Models (LLMs) have become central to modern software development, yet they often function as "black boxes," making it difficult to understand the logic behind their generated code. While In-Context Learning (ICL) allows developers to guide these models using demonstration examples, the process of selecting effective "prototypes" remains largely unprincipled. Protocode addresses these challenges by introducing a framework for intelligent prototype selection and syntax-aware interpretability.
Methodology
Protocode employs a two-stage approach to improve both the functional correctness and the transparency of code generation.
Manifold-Based Prototype Sampling
Rather than relying on simple similarity metrics, Protocode uses a neural network to map code embeddings onto a lower-dimensional manifold. By utilizing piecewise-linear manifold learning and Proxy-Anchor metric learning, the system identifies prototypes that are both geometrically representative and semantically distinct across various programming tasks.
AST-Grounded Interpretability
To explain model outputs, Protocode calculates the gradient of the similarity between sampled prototypes and generated tokens. These attribution scores are then mapped onto an Abstract Syntax Tree (AST). This process groups individual tokens into high-level syntax categories, such as "Decisions," "Iterations," and "Exceptions," providing a clear "confidence map" of the model's logic.
Evaluation
The framework was evaluated using the Magicoder-OSS-Instruct-75K dataset as a source for prototypes and the MBPP and MBPP+ benchmarks for performance testing. The study included a diverse range of LLMs, including Qwen, Llama, Falcon, and StarCoder.
Results
Experimental results demonstrate that Protocode consistently outperforms standard sampling strategies, such as diversity-based or simple similarity-based selection.
As shown in the results, Protocode significantly improved the performance of smaller, code-specific models. For instance, the Qwen2.5-coder-0.5B model achieved a pass@10 of 0.122 using Protocode, compared to 0.116 for the base model. Similar trends were observed in the Codellama-7B model.
Discussion and Insights
The AST-based analysis revealed that models exhibit varying "confidence profiles." While most models are highly proficient in handling Data Structures and Scope, they frequently struggle with Exception Handling and complex control flows. This suggests that while models are dependable for structural code, they require more guidance or better prototypes for logic-intensive segments. Furthermore, by utilizing gradients rather than full vocabulary distributions, Protocode achieves this level of interpretability with significantly lower memory overhead than previous state-of-the-art methods.
Conclusion
Protocode provides a robust framework for enhancing LLM-based code generation through principled prototype selection and efficient, syntax-aware explanations. Future research will explore "prototype steering," leveraging these interpretability insights to actively direct models toward specific coding patterns or safety standards.
Summary updated