TL;DR
QERA (Question-driven Evidence Reasoning Architecture) is a new inference-time method that helps LLMs reason over messy, real-life data like group chats and activity logs. By "interviewing the context" to uncover implicit requirements before writing a response, QERA significantly boosts the performance of frontier models like GPT-6 and Grok 4.7 on complex tasks while remaining more token-efficient than existing state-of-the-art baselines.
What Problem Does This Paper Address?
When we ask an AI assistant to summarize a long Slack thread or analyze months of fitness logs, we often assume it remembers every rule we’ve ever set. However, the authors of this paper highlight a major gap: the request is not the specification.
In real-world "context learning," the final user query is often just the tip of the iceberg. Crucial constraints—like "don't use technical jargon" or "only include data from Tuesday"—might have been mentioned several turns ago or are buried deep in the provided documents. The authors found that over 80% of evaluation rubrics for these tasks are "implicit," meaning they aren't stated in the final request. Standard LLMs fail these implicit rules frequently, passing only 67% of them compared to 81% of explicit ones.
Methods
QERA introduces a four-stage "ask-before-you-answer" pipeline that operates entirely at inference time, requiring no extra training or task-specific demonstrations.
- Ask: Before generating a response, the LLM creates a Requirement Checklist (what the reply must deliver) and Information Needs (what specific facts must be found in the context). Deterministic code filters ensure these questions stay faithful to the user's actual words.
- Answer: The LLM writes a draft guided by the checklist and a "fixed answer protocol"—a set of strict rules for attribution, formatting, and handling updates or corrections found in the text.
- Check: The system re-asks the initial questions against the draft. It uses code to "audit" the response, verifying that quotes actually exist in the source text and that numeric totals add up.
- Repair: If the check finds errors, a final call patches the draft. This is protected by a Guard that rejects the repair if it's too short, cut off, or introduces data from outside the conversation.
Datasets
The researchers evaluated QERA using CL-bench Life, a challenging benchmark consisting of 405 tasks and over 5,000 human-written rubrics. These tasks involve reasoning over "web traces" like:
- Communication: Group chats and meeting transcripts.
- Behavioral Records: Game logs (e.g., Chess.com), fitness data, and digital footprints.
- Fragmented Info: Document revision histories and personal archives.
Key Insights and Ideas
- The "Aha" Moment: Most of the performance gain happens before the final answer is even written. By forcing the model to plan its information needs, it "reads" the context more effectively.
- Implicit vs. Explicit: LLMs are already quite good at following instructions written directly in the prompt. The real frontier is recovering the "hidden" instructions scattered throughout a long history.
- Self-Correction is Hard, Planning is Easier: The authors found that while LLMs struggle to fix their own reasoning errors after the fact, they are much better at following a structured plan they generated themselves.
- Code as a Safety Rail: QERA uses deterministic code to handle things LLMs are notoriously bad at, such as verifying exact string matches for quotes and performing basic arithmetic audits.
Key Results
QERA achieved the highest solving rate across all four frontier LLMs tested (Kimi K3, Grok 4.7, GLM-5, and GPT-6-Luna).
- Solving Rate: QERA averaged a 21.0% solving rate, a significant jump over direct "Vanilla" answering (14.6%) and the strongest previous baseline, PSCI (18.2%).
- Implicit Success: The gains were almost entirely concentrated on implicit rubrics, where QERA outperformed the best baseline by 2.5 percentage points.
- Efficiency: While QERA uses about 5x the tokens of a single call, it uses only 4-5% of the tokens required by Ctx2Skill, another high-performing method that relies on expensive multi-agent self-play.
Limitations and Future Work
- LLM Dependency: QERA is only as good as the underlying model. If the LLM hallucinates a requirement during the "Ask" phase, that error will likely persist into the final answer.
- Cost: Although cheaper than some baselines, it is still significantly more expensive than a single model call. The authors suggest that the "Check" stage could be made optional to save costs.
- Text-Only: The current study focuses on text. Extending QERA to multimodal contexts—where an assistant might need to "ask" questions about screenshots or voice notes—is a logical next step.
- Judge Reliability: The evaluation relies on an LLM judge (GPT-5.1), which, while validated against humans, can still make mistakes on highly nuanced rubrics.
Summary updated