When building LLM-powered agents, the default pattern in most frameworks (like LangChain or OpenAI’s Assistant API) is to feed tool outputs directly back into the model’s context. The model then "re-emits" this data to the user.
A new paper from Tung Hoang at AwardWise reveals a critical flaw in this pattern: Fidelity Collapse. As the amount of structured data (like a table of flight results) grows, the model’s ability to accurately re-emit that data doesn't just degrade—it falls off a cliff.
TL;DR
LLM agents that re-generate structured data from their own context suffer a "fidelity collapse" as table sizes increase, regardless of the model's reasoning capability. By switching to a BYPASS architecture—where the model only generates filter arguments and data is delivered via a typed side channel—developers can eliminate this corruption while reducing costs by up to 200x.
What Problem Does This Paper Address?
The authors identify a phenomenon called structured hallucination. Unlike typical hallucinations where a model invents facts, structured hallucinations occur when a model retrieves perfectly correct data from a tool but corrupts the values while formatting them into JSON or text.
Current benchmarks often miss this because they check if the JSON is valid, not if the content is correct. In a multi-turn agent, this problem compounds: if the model corrupts a table in Turn 1, it uses that "poisoned" output as the input for Turn 2, leading to a total breakdown in reliability.
Methods
The researchers conducted a controlled A/B test on a production award-travel agent (AwardWise). They compared two delivery architectures over identical "frozen" data payloads:
- SYNTHESIS (The Status Quo): The agent retrieves full record sets into its context, filters them "in-head," and re-emits the results.
- BYPASS (The Proposed Fix): The agent keeps records in a separate session state. The model only translates the user's request into deterministic filter arguments (e.g.,
max_price: 500). A traditional code-based evaluator then filters the state and sends the results to the UI via a "typed side channel," bypassing the model's generation entirely.
To isolate the effect of table size, they used content-controlled prefix ladders, where they fed the model the first K rows of a fixed dataset and measured how often it could correctly filter and re-emit them.
Datasets
The study used a production-grade corpus of 408 captures containing 9,809 records of real flight data (from Seats.aero and SerpAPI). They evaluated the architectures across 24 unique dialogues (339 scored turns) using four different model families: Gemini 2.5 Flash/Pro, GPT-4o-mini, and Claude Haiku 4.5.
Key Insights and Ideas
- Size, Not Depth, is the Killer: The primary driver of failure isn't the number of conversation turns, but the number of rows the model has to process.
- The "Capability Threshold": Every model has a "collapse threshold." For GPT-4o-mini, fidelity collapses at fewer than 10 rows. For Gemini 2.5 Pro, the threshold is higher (~198 rows), but the downward trend is inevitable.
- Architecture Trumps Scaling: While "renting" a smarter model (moving from Flash to Pro) improves results, it is an expensive band-aid. The BYPASS architecture provides "immunity" to size-based collapse across all models.
- The Cost of Error: In the SYNTHESIS model, you pay for every token the model re-emits. When you factor in the cost of failed turns, the price per correct answer sky-rockets.
Key Results
- Accuracy Gap: Across the full corpus, BYPASS achieved 92.3% turn fidelity, while SYNTHESIS managed only 45.7%.
- The Collapse: In the prefix ladder tests, SYNTHESIS fidelity for Gemini Flash dropped from near-perfect at 5 rows to under 8% once the table exceeded 74 rows. BYPASS remained flat at ~100% regardless of size.
- Economic Impact: Beyond 100 rows, the cost per correct answer for SYNTHESIS reached $2.53**, compared to a flat **$0.012 for BYPASS—a 200x difference in efficiency.
Limitations and Future Work
The authors acknowledge that the BYPASS architecture's advantage is "joint"—it combines bypassing the LLM for delivery with preserving structured state. They did not test a hybrid version (state preservation + LLM delivery) because it doesn't match common deployment patterns.
Future work could explore whether this decay varies with "record width" (number of fields per row) or if specific prompt engineering techniques can push the collapse threshold further back. However, the core message for engineers is clear: If you want your agent to handle data at scale, keep the data out of the model's mouth.
Summary updated