Introduction
Modern software development increasingly relies on Large Code Models (LCMs) for productivity. While most research focuses on "API Recommendation"—predicting the next method call after a trigger—this narrow view fails to capture the full complexity of developer needs. This paper introduces a more comprehensive framework by evaluating LCMs across three distinct real-world scenarios: determining when to call an API, which specific API to select from a library, and how to correctly provide its arguments.
Methodology and Contextual Enrichment
The researchers evaluated nine prominent LCMs, including StarCoder, CodeLlama, and DeepSeek-Coder, with parameter counts ranging from 1.3B to 34B. A central hypothesis of the study is that the "context" provided in a prompt significantly influences model performance. To test this, the authors defined several context types to enrich the model's input:
- Base (Function Context): The immediate code leading up to the API call.
- File Context (F): Code preceding the current function.
- Function Comment (C): Natural language descriptions of the method's purpose.
- Suffix Context (S): Code following the API call, utilizing Fill-In-the-Middle (FIM) capabilities.
- Import Messages (I): Library imports at the top of the file.
- Library Candidates (L): A curated list of available APIs to mitigate hallucinations.
The Spring-API Benchmark
To support this granular evaluation, the authors developed a new benchmark derived from 853 popular Java projects on GitHub. Focusing on the ubiquitous Spring Framework, the dataset includes 4,146 API usage entries across 3,136 files. Crucially, these entries are categorized into the "When, Which, and How" scenarios, allowing for a detailed analysis of model strengths and weaknesses across different stages of the development workflow.
Key Insights and Results
The "When" Difficulty Gap
The study reveals a significant performance disparity between scenarios. Models achieved an average Exact Match (EM) score of 64.04% for "How to use" but plummeted to 30.43% for "When to use." This 34% gap suggests that while models are adept at syntax and argument filling, they struggle to reason about the high-level logic required to place an API call correctly.
Context as a Force Multiplier
One of the most striking findings is that context can compensate for model size. A small DeepSeek-Coder 1.3B model, when provided with full context (imports, comments, and suffix), can outperform a DeepSeek-Coder 33B model that only sees the local function. Specifically, in the challenging "When to use" scenario, adding all context types improved the average EM score by 89.2% (from 30.43% to 57.57%).
The Throughput Tax
Enriching prompts is not without cost. Incorporating all studied context types increases prompt length by approximately 335%, which in turn slashes model throughput by 54%. This creates a significant engineering challenge for developers of real-time IDE plugins, where latency is a critical factor for user experience.
Conclusion
While LCMs show great promise, their ability to decide "when" to use an API remains a bottleneck. Future research should look beyond single-file context to explore cross-file dependencies and evaluate performance across different programming languages and frameworks. Additionally, the trade-off between contextual richness and inference speed remains a vital area for optimization in production environments.
Summary updated