Adapting LLMs for Structured Natural Language API Integration

Robin Chan, Katsiaryna Mirylenka, Thomas Gschwind, Christoph Miksovic, Paolo Scotton, Enrico Toniato, Abdel LabbiEMNLP 2024

TL;DR

This paper tackles the challenge of seamlessly integrating diverse software APIs using natural language. They propose a novel approach that fine-tunes Large Language Models (LLMs) on automatically generated API flow data, significantly improving their ability to understand user intent and construct complex API sequences compared to traditional methods.

What Problem Does This Paper Address?

Integrating different software services is essential for automating workflows, but the sheer variety and complexity of APIs make this a difficult task. Existing methods often rely on techniques like Named Entity Recognition (NER) and knowledge graphs, which struggle to generate sophisticated control flow structures like conditional logic or loops. This requires users to have deep technical knowledge of the APIs or to manually search through service catalogs. The authors aim to bridge this gap by enabling users to express their desired integrations using natural language, allowing LLMs to automatically generate the necessary API calls.

Methods

The core of the proposed solution lies in adapting LLMs for API integration. The authors introduce a two-pronged approach:

  1. Synthetic Data Generation: They automatically generate training data by leveraging OpenAPI specifications. This involves extracting descriptions from API definitions and creating operation-specific templates. To enhance data variability and improve generalization, they augment this data with paraphrasing. This synthetic data captures the building blocks of API flows, including single API calls, sequential actions, conditional logic, and loops.
  2. Constrained Decoding: At inference time, they optionally enforce syntactic and schema adherence using constrained decoding. This technique restricts the LLM's output to valid API calls and flow structures, preventing the generation of syntactically incorrect or nonsensical sequences. They adapt existing constrained decoding methods for beam search to improve efficiency.

Datasets

The paper uses a combination of existing and newly generated data:

  • Training Data: A large dataset of 55,000 samples was generated synthetically using OpenAPI specifications.
  • Evaluation Data:
    • In-Distribution (ID) Test Set: A 7,000-sample split of the generated data.
  • Out-of-Distribution (OOD) Test Suite: A custom-designed benchmark to evaluate generalization. This suite includes scenarios like common app name variations, omitting application or object names from utterances, complex flow structures (more than two actions), and user-collected integration flows requiring intricate reasoning.

Key Insights and Ideas

  • LLMs can implicitly learn API structure: Fine-tuning LLMs on automatically generated API flows allows them to learn structural constraints without explicit programming of these rules.
  • Synthetic data is crucial: Generating diverse and representative synthetic data is key to training LLMs that can handle the complexity of API integration.
  • Constrained decoding significantly boosts precision: While LLMs can learn API structures, enforcing them with constrained decoding dramatically improves the accuracy of generated API calls, especially in ambiguous or out-of-distribution scenarios.
  • NER struggles with complex flows: Traditional NER-based approaches are limited in their ability to handle complex API flows and variations in user phrasing, highlighting the advantage of LLMs.
  • Conversational LLMs offer an edge: Models fine-tuned for conversational interactions (like LLaMA) show promise in handling ambiguous user requests by allowing for interactive clarification.

Key Results

The paper demonstrates that their fine-tuned LLM approach significantly outperforms NER-based baselines and retrieval-augmented generation (RAG) methods.

  • In-Distribution Performance: Fully tuned BLOOM-3B with constrained decoding achieved an Exact Matching (EM) score of 85.2% and a Triplet Precision (TP) of 100.0%. Fully tuned LLaMA-13B achieved an EM of 92.1% and TP of 89.5%.
  • Out-of-Distribution Performance: The same models also showed substantial improvements in OOD scenarios. Fully tuned BLOOM-3B with CD achieved an EM of 40.6% and TP of 100.0%. Conversational LLaMA-13B achieved an EM of 57.6% and TP of 91.7%.
  • Constrained Decoding Impact: Constrained decoding consistently improved triplet precision across models, particularly for OOD samples, indicating better adherence to API definitions.

Limitations and Future Work

  • Argument Generation: The current approach focuses on generating API call sequences (triplets) and does not generate arguments for these calls. The authors acknowledge that generating arguments from natural language is challenging due to the tediousness of specifying them in utterances.
  • Application Ambiguity: While the model handles some ambiguity, accurately deducing the correct application when it's missing from the utterance can still be challenging.
  • Future Work: The authors suggest exploring semi-supervised approaches for argument generation and developing more efficient constrained decoding techniques, especially for interactive API generation with models like LLaMA. They also plan to investigate the potential of multi-turn prompting for handling ambiguous user requests more effectively.

Summary updated

BibTeX
@inproceedings{chan2024adapting,
  author = {Robin Chan and Katsiaryna Mirylenka and Thomas Gschwind and Christoph Miksovic and Paolo Scotton and Enrico Toniato and Abdel Labbi},
  title = {{Adapting LLMs for Structured Natural Language API Integration}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure