TL;DR
This paper introduces a novel behavior-driven evaluation framework for Compound AI (CAI) systems, moving beyond traditional benchmarks to assess real-world operational performance. By generating explicit test specifications that mimic user interactions and implementing them as conversational QA pairs, the framework effectively uncovers failure modes missed by existing methods, demonstrating significantly higher failure rates in targeted scenarios.
What Problem Does This Paper Address?
Compound AI (CAI) systems, which integrate Large Language Models (LLMs) with tools and retrievers, are increasingly used in information-seeking applications. However, evaluating their true operational performance is challenging. Traditional benchmark datasets and aggregate metrics often fail to capture how these systems behave in realistic, multi-step scenarios, leading to a false sense of security. Direct user evaluation is scalable, and existing synthetic methods often fail to reflect real-world usage or overlook system-level behaviors. The authors aim to bridge this gap by developing a method that probes CAI system behaviors across a spectrum of simulated scenarios to identify potential failure modes.
Methods
The proposed framework operates in two phases:
- Test Specification Generation: This phase uses LLMs to generate a large pool of "behavior-driven test specifications." These specifications are explicit descriptions of expected system behaviors in specific scenarios, following a Given-When-Then format. Submodular optimization is then used to select a subset of these specifications that maximizes both semantic diversity and document coverage. This approach aims to create targeted test data that is otherwise hard to acquire.
- Test Specification Implementation: The selected specifications are transformed into document-grounded conversational question-answer (QA) pairs. This involves:
- Document Extraction: Retrieving relevant documents (text, policy, and tabular) based on the test specification.
- Graph Construction: Modeling document relationships through type-specific graphs (text-text, table-table, or a unified graph).
- Multi-Document Chain Extraction: Extracting chains of related documents using Depth-First Search (DFS) on the constructed graph.
- Conversation Generation: Prompting an LLM with the system description, test specification, and extracted document context to generate multi-turn conversations.
- Verification: A three-stage filtering process (retrieval-augmented correction, independent LLM verification, and human verification) ensures the quality and adherence of the generated QA pairs.
Datasets
The authors evaluated their framework on existing datasets:
- QuAC (Question Answering in Context): Used for evaluating the Celebrity Trivia Assistant.
- HybriDialogue: Used for evaluating the Sports Trivia Assistant.
- Airline Assistant (LangGraph): Used for evaluating the Airline Assistant.
The framework also utilizes document collections associated with these datasets for grounding the generated conversations.
Key Insights and Ideas
- Behavior-Driven Evaluation is Crucial: Traditional benchmarks, while useful for component-level evaluation, mask system-level failures. Focusing on explicit behaviors and scenarios is key to uncovering real-world limitations.
- Submodular Optimization for Diversity: Using submodular optimization for selecting test specifications effectively balances semantic diversity and document coverage, leading to more comprehensive and targeted evaluations.
- LLMs as Behavior Simulators: LLMs can be effectively used to simulate human behavior and generate realistic test specifications and conversational QA pairs, reducing reliance on costly human annotation.
- Failure Modes are Scenario-Specific: The framework highlights that CAI systems can perform well on average but exhibit significant failure rates in specific, complex scenarios (e.g., multi-hop reasoning, coreference resolution, tool orchestration).
- LLMs as Reliable Evaluators: LLM-based evaluation (LLM-as-a-judge) shows strong statistical correlation with human judgments, making it a scalable and reliable method for assessing CAI system performance.
Key Results
- The proposed framework identified failure modes missed by traditional metrics, demonstrating failure rates twice as high as human-curated datasets in targeted scenarios.
- For the Airline Assistant, scenarios like "Travel Information for a Multi-City Trip" and "Flight Status across Multiple Timezones" showed failure rates of 78.0% and 78.6%, respectively, indicating systematic issues in tool orchestration.
- The Sports Trivia Assistant, in the "Significance of a Sports Award" scenario, reached a 71.64% failure rate, more than double the benchmark rate of 26.1%.
- The framework's selected test specifications achieved higher diversity and document coverage compared to baseline selection methods (Random, Max-Min), confirmed by human evaluation.
- LLM evaluators (gpt-40 and Claude 3.5 Sonnet) showed statistically significant positive correlations with human assessments, indicating their reliability.
- The framework's generated conversations were rated highly for correctness, answerability, and coherence, comparable to human-authored data.
Limitations and Future Work
- Human Preference Integration: While the framework optimizes for diversity and document coverage, future work could incorporate human preferences to further enhance scenario relevance.
- Complex Policy Queries: The current framework does not explicitly handle complex policy-related queries that might involve reasoning over different types of documents.
- Scalability of Human Verification: While LLM-based evaluation is scalable, human verification remains crucial for high-stakes scenarios, and its scalability is a consideration.
- Model Overlap: The authors investigated potential self-referential bias when the LLM used for generation is also the core LLM of the CAI system. They found minimal impact on failure rates, suggesting the approach is robust to this specific form of overlap.
Summary updated