TL;DR
WAR-CRAFT is a recommender-agnostic framework that refines Web API recommendation sets by auditing them against actual provider documentation. By treating refinement as a structured process of deleting redundant APIs, auditing requirement coverage, and selectively adding missing services, it significantly boosts recommendation accuracy and halves functional redundancy across diverse upstream models.
What Problem Does This Paper Address?
Building modern "mashup" applications involves selecting a set of Web APIs to fulfill multiple functional requirements. While many recommenders exist—ranging from collaborative filtering to modern LLM-based agents—they often fail at the "set-level." Specifically, they frequently recommend APIs that don't actually support the requested features, include redundant APIs that do the same thing, or leave critical requirements unaddressed.
The core issue is that these recommenders rarely check their outputs against the "ground truth": the provider's own documentation. Furthermore, blindly adding APIs to fill every perceived gap can be counterproductive, as some requirements might not need a separate API or might not have a suitable provider in the catalog.
Methods
The authors propose WAR-CRAFT, a post-recommendation refinement framework that acts as an "auditor" for any frozen upstream recommender. It operates in three main stages:
- Set-Aware Filtering (Role Writer): Instead of looking at APIs in isolation, an LLM evaluates the entire recommended set jointly. It removes APIs that are unsupported by documentation or whose functions are entirely subsumed by another API in the set. For every API it keeps, it writes a concise "functional role" describing its specific contribution.
- Joint Requirement Auditing: The system generates a checklist of requirements based on the user's request. It then audits the retained APIs against this checklist. If a requirement is already covered, it's marked as such; if not, it's flagged as a gap.
- Catalog-Constrained Resolution & Verification: For each missing requirement, a fine-tuned Resolver selects a new API from the catalog or chooses to abstain (a critical "NO_ADD" option). Finally, a Commit Verifier double-checks these new proposals against their documentation to ensure they actually do what the Resolver thinks they do.
Datasets
The study uses the ProgrammableWeb corpus, a standard benchmark in the field, containing 8,217 mashups and 1,647 APIs. The authors evaluated WAR-CRAFT by refining the outputs of seven different "upstream" recommenders, including traditional ranking models (like WARBERT) and generative LLM-based ones (like LLMAR).
Key Insights and Ideas
- "Adding is a decision in its own right": One of the most important findings is that always trying to fill every gap lowers performance. The Resolver's ability to abstain (returning
<NO_ADD>) when no suitable API exists or when a requirement is better handled elsewhere is vital for maintaining a high-quality, compact set. - Documentation as the Anchor: By grounding every "KEEP," "DELETE," or "ADD" decision in verbatim evidence from API documentation, the framework mitigates the "hallucination" problems common in LLM-based recommendation.
- Joint Evaluation Reduces Bloat: Evaluating redundancy at the set level (rather than per-item) allowed the system to significantly reduce the integration effort for developers by removing unnecessary duplicate services.
- Recommender-Agnosticism: Because it treats the upstream recommender as a "black box," WAR-CRAFT can be layered on top of any existing or future recommendation technology without needing to retrain the base model.
Key Results
- Accuracy Boost: Across seven different recommenders, WAR-CRAFT raised the mean SetF1 score from 0.558 to 0.644 (an 8.64 percentage point gain).
- Redundancy Reduction: The framework roughly halved functional redundancy, dropping it from 0.127 to 0.065.
- Efficiency Gains: The average size of the recommended sets decreased from 1.86 to 1.59 APIs, making the recommendations more concise and easier for developers to integrate.
- Human-Verified Quality: Human audits confirmed that the generated functional roles were highly accurate, with 89.9% of ratings receiving the maximum score for alignment, grounding, and clarity.
Limitations and Future Work
- One-Pass Auditing: Currently, the system performs a single refinement pass. It does not re-audit the final set to see if the newly added APIs introduced new redundancies or gaps.
- Complex Decompositions: If the Auditor fails to properly break down a complex, multi-part requirement, the Resolver (which picks one API per query) might struggle to find a complete solution.
- LLM Memorization: While the authors took steps to ensure the LLM was judging based on provided documentation, there is always a risk that the model "remembered" popular APIs from its pre-training data.
Future work could explore iterative auditing cycles or more sophisticated requirement decomposition strategies to handle even more complex application requests.
Summary updated