WAR-CRAFT: Refining Web API Recommendations through Requirement Auditing

Zishuo Xu, Dezhong Yao, Zhiqiang XuarXiv 2026 · arXiv

TL;DR

WAR-CRAFT is a recommender-agnostic framework that refines Web API recommendation sets by auditing them against actual provider documentation. By treating refinement as a structured process of deleting redundant APIs, auditing requirement coverage, and selectively adding missing services, it significantly boosts recommendation accuracy and halves functional redundancy across diverse upstream models.

What Problem Does This Paper Address?

Building modern "mashup" applications involves selecting a set of Web APIs to fulfill multiple functional requirements. While many recommenders exist—ranging from collaborative filtering to modern LLM-based agents—they often fail at the "set-level." Specifically, they frequently recommend APIs that don't actually support the requested features, include redundant APIs that do the same thing, or leave critical requirements unaddressed.

The core issue is that these recommenders rarely check their outputs against the "ground truth": the provider's own documentation. Furthermore, blindly adding APIs to fill every perceived gap can be counterproductive, as some requirements might not need a separate API or might not have a suitable provider in the catalog.

Methods

The authors propose WAR-CRAFT, a post-recommendation refinement framework that acts as an "auditor" for any frozen upstream recommender. It operates in three main stages:

  1. Set-Aware Filtering (Role Writer): Instead of looking at APIs in isolation, an LLM evaluates the entire recommended set jointly. It removes APIs that are unsupported by documentation or whose functions are entirely subsumed by another API in the set. For every API it keeps, it writes a concise "functional role" describing its specific contribution.
  2. Joint Requirement Auditing: The system generates a checklist of requirements based on the user's request. It then audits the retained APIs against this checklist. If a requirement is already covered, it's marked as such; if not, it's flagged as a gap.
  3. Catalog-Constrained Resolution & Verification: For each missing requirement, a fine-tuned Resolver selects a new API from the catalog or chooses to abstain (a critical "NO_ADD" option). Finally, a Commit Verifier double-checks these new proposals against their documentation to ensure they actually do what the Resolver thinks they do.
Figure 2: Overview of the WAR-CRAFT framework, showing the pipeline from initial recommendation to the final refined set.

Datasets

The study uses the ProgrammableWeb corpus, a standard benchmark in the field, containing 8,217 mashups and 1,647 APIs. The authors evaluated WAR-CRAFT by refining the outputs of seven different "upstream" recommenders, including traditional ranking models (like WARBERT) and generative LLM-based ones (like LLMAR).

Key Insights and Ideas

  • "Adding is a decision in its own right": One of the most important findings is that always trying to fill every gap lowers performance. The Resolver's ability to abstain (returning <NO_ADD>) when no suitable API exists or when a requirement is better handled elsewhere is vital for maintaining a high-quality, compact set.
  • Documentation as the Anchor: By grounding every "KEEP," "DELETE," or "ADD" decision in verbatim evidence from API documentation, the framework mitigates the "hallucination" problems common in LLM-based recommendation.
  • Joint Evaluation Reduces Bloat: Evaluating redundancy at the set level (rather than per-item) allowed the system to significantly reduce the integration effort for developers by removing unnecessary duplicate services.
  • Recommender-Agnosticism: Because it treats the upstream recommender as a "black box," WAR-CRAFT can be layered on top of any existing or future recommendation technology without needing to retrain the base model.

Key Results

  • Accuracy Boost: Across seven different recommenders, WAR-CRAFT raised the mean SetF1 score from 0.558 to 0.644 (an 8.64 percentage point gain).
  • Redundancy Reduction: The framework roughly halved functional redundancy, dropping it from 0.127 to 0.065.
  • Efficiency Gains: The average size of the recommended sets decreased from 1.86 to 1.59 APIs, making the recommendations more concise and easier for developers to integrate.
  • Human-Verified Quality: Human audits confirmed that the generated functional roles were highly accurate, with 89.9% of ratings receiving the maximum score for alignment, grounding, and clarity.
Figure 1: Post-recommendation refinement for a mashup request. The unsupported API B is removed, and the uncovered map-display and email-delivery requirements are filled with catalog APIs C and D after verification against their documentation. Each selected API receives a functional role describing its contribution.

Limitations and Future Work

  • One-Pass Auditing: Currently, the system performs a single refinement pass. It does not re-audit the final set to see if the newly added APIs introduced new redundancies or gaps.
  • Complex Decompositions: If the Auditor fails to properly break down a complex, multi-part requirement, the Resolver (which picks one API per query) might struggle to find a complete solution.
  • LLM Memorization: While the authors took steps to ensure the LLM was judging based on provided documentation, there is always a risk that the model "remembered" popular APIs from its pre-training data.

Future work could explore iterative auditing cycles or more sophisticated requirement decomposition strategies to handle even more complex application requests.

Summary updated

Abstract

Web API recommenders return a set of APIs for a mashup request but do not check it against provider documentation, the main public evidence of an API's capabilities. The set can therefore contain unsupported or redundant APIs and miss requested capabilities. Filling every apparent gap does not fix this, because a gap may already be served, may not need its own API, or may have no documented provider. We study post-recommendation refinement, which revises a frozen recommender's output using only the request, the catalog, and provider documentation. We propose WAR-CRAFT, which treats refinement as three decisions. An LLM removes unsupported or subsumed APIs by judging candidates jointly, then audits the remaining set against a requirement checklist derived from the request. For each uncovered requirement, a fine-tuned catalog-constrained Resolver proposes a catalog API or abstains, and an independent documentation check admits only supported proposals. On ProgrammableWeb with 7 heterogeneous upstream recommenders, WAR-CRAFT raises mean SetF1 from 0.558 to 0.644, above the strongest of 5 refinement baselines (62.5%), and roughly halves functional redundancy (0.127 to 0.065) without changing mean functional coverage. Gains range from +2.2 points on generative recommenders to +18.2 on fixed-cutoff rankers. At matched set sizes, WAR-CRAFT still outperforms truncating the upstream output, so the gain is not explained by set-size calibration alone. Our code is available at https://github.com/ZsXu-enrico/WAR-CRAFT.
BibTeX
@misc{xu2026war,
  author = {Zishuo Xu and Dezhong Yao and Zhiqiang Xu},
  title = {{WAR-CRAFT: Refining Web API Recommendations through Requirement Auditing}},
  year = {2026},
  eprint = {2611.00564},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00564},
}
Figure