Ask Before You Answer: A Question-Driven Evidence Reasoning Architecture (QERA) for In-Context Learning

Jinhu Qi, Minda Hu, Wentao Zhang, Zixing Song, Irwin King

TL;DR

QERA (Question-driven Evidence Reasoning Architecture) is a new inference-time method that helps LLMs reason over messy, real-life data like group chats and activity logs. By "interviewing the context" to uncover implicit requirements before writing a response, QERA significantly boosts the performance of frontier models like GPT-6 and Grok 4.7 on complex tasks while remaining more token-efficient than existing state-of-the-art baselines.

What Problem Does This Paper Address?

When we ask an AI assistant to summarize a long Slack thread or analyze months of fitness logs, we often assume it remembers every rule we’ve ever set. However, the authors of this paper highlight a major gap: the request is not the specification.

In real-world "context learning," the final user query is often just the tip of the iceberg. Crucial constraints—like "don't use technical jargon" or "only include data from Tuesday"—might have been mentioned several turns ago or are buried deep in the provided documents. The authors found that over 80% of evaluation rubrics for these tasks are "implicit," meaning they aren't stated in the final request. Standard LLMs fail these implicit rules frequently, passing only 67% of them compared to 81% of explicit ones.

Methods

QERA introduces a four-stage "ask-before-you-answer" pipeline that operates entirely at inference time, requiring no extra training or task-specific demonstrations.

  1. Ask: Before generating a response, the LLM creates a Requirement Checklist (what the reply must deliver) and Information Needs (what specific facts must be found in the context). Deterministic code filters ensure these questions stay faithful to the user's actual words.
  2. Answer: The LLM writes a draft guided by the checklist and a "fixed answer protocol"—a set of strict rules for attribution, formatting, and handling updates or corrections found in the text.
  3. Check: The system re-asks the initial questions against the draft. It uses code to "audit" the response, verifying that quotes actually exist in the source text and that numeric totals add up.
  4. Repair: If the check finds errors, a final call patches the draft. This is protected by a Guard that rejects the repair if it's too short, cut off, or introduces data from outside the conversation.
Figure 2: Answering directly (left) misses what a request leaves unsaid: the four LLMs of our study, answering directly, pass 81% of the rubrics whose wording the request largely contains but only 67% of the others. QERA (right) first asks what the reply must deliver (checklist 𝑂) and what must be found in the context (needs 𝑁); code filters Φ keep these questions 𝑄faithful to the user’s words. The reply is drafted from the whole conversation under 𝑄; the check asks 𝑄again and returns evidence 𝐸 with quotes, which code audits; a local repair is kept only if the guard Γ accepts it. Robots count LLM calls (one frozen LLM throughout); dashed boxes are code.

Datasets

The researchers evaluated QERA using CL-bench Life, a challenging benchmark consisting of 405 tasks and over 5,000 human-written rubrics. These tasks involve reasoning over "web traces" like:

  • Communication: Group chats and meeting transcripts.
  • Behavioral Records: Game logs (e.g., Chess.com), fitness data, and digital footprints.
  • Fragmented Info: Document revision histories and personal archives.

Key Insights and Ideas

  • The "Aha" Moment: Most of the performance gain happens before the final answer is even written. By forcing the model to plan its information needs, it "reads" the context more effectively.
  • Implicit vs. Explicit: LLMs are already quite good at following instructions written directly in the prompt. The real frontier is recovering the "hidden" instructions scattered throughout a long history.
  • Self-Correction is Hard, Planning is Easier: The authors found that while LLMs struggle to fix their own reasoning errors after the fact, they are much better at following a structured plan they generated themselves.
  • Code as a Safety Rail: QERA uses deterministic code to handle things LLMs are notoriously bad at, such as verifying exact string matches for quotes and performing basic arithmetic audits.
Figure 1: A CL-bench Life task (Game Logs). The user’s first turn rules out chess notation; the request two turns later does not repeat it. Answering directly, all four LLMs write chess notation. QERA’s checklist carries the rule forward, and QERA solves the task with Kimi K3 and Grok 4.7 (both replies shown are from Kimi K3).

Key Results

QERA achieved the highest solving rate across all four frontier LLMs tested (Kimi K3, Grok 4.7, GLM-5, and GPT-6-Luna).

  • Solving Rate: QERA averaged a 21.0% solving rate, a significant jump over direct "Vanilla" answering (14.6%) and the strongest previous baseline, PSCI (18.2%).
  • Implicit Success: The gains were almost entirely concentrated on implicit rubrics, where QERA outperformed the best baseline by 2.5 percentage points.
  • Efficiency: While QERA uses about 5x the tokens of a single call, it uses only 4-5% of the tokens required by Ctx2Skill, another high-performing method that relies on expensive multi-agent self-play.
QERA sits on the accuracy-cost Pareto front, offering better performance than PSCI with fewer tokens.

Limitations and Future Work

  • LLM Dependency: QERA is only as good as the underlying model. If the LLM hallucinates a requirement during the "Ask" phase, that error will likely persist into the final answer.
  • Cost: Although cheaper than some baselines, it is still significantly more expensive than a single model call. The authors suggest that the "Check" stage could be made optional to save costs.
  • Text-Only: The current study focuses on text. Extending QERA to multimodal contexts—where an assistant might need to "ask" questions about screenshots or voice notes—is a logical next step.
  • Judge Reliability: The evaluation relies on an LLM judge (GPT-5.1), which, while validated against humans, can still make mistakes on highly nuanced rubrics.

Summary updated

Abstract

AI assistants are increasingly asked to reason over a user's own web traces: group chats, community threads, activity logs and revision histories. On CL-bench Life, a benchmark of such real-life context learning, frontier LLMs solve fewer than one task in five. We find that the request is not the specification: over 80% of the benchmark's rubrics share less than 60% of their content words with the final request, as for a rule set two turns earlier or every record that meets the user's criterion, and LLMs answering directly pass only 67% of these implicit rubrics against 81% of the others. We propose QERA, a Question-driven Evidence Reasoning Architecture that asks before it answers. QERA first turns the request into a requirement checklist and typed information needs, filtered by code for faithfulness to the user's words; it then answers from the whole conversation, asks the same questions again with quotes verified by code, and applies a guarded local repair. It needs no training or rubrics and makes about four LLM calls per task. On all 405 tasks with four frontier LLMs, QERA attains the highest solving rate on every LLM (21.0% on average, against 18.2% for the strongest of three inference-time baselines and 14.6% for direct answering) and leads every baseline significantly when pooled across LLMs, while using fewer tokens than PSCI and 4-5% of the tokens of Ctx2Skill. Ablations on GPT-6-Luna show that the gain is settled before the answer, by a fixed answer protocol (+3.8 rubric points) and the request-specific questions (+1.6), and that it is concentrated on implicit rubrics, with checking afterwards refining mainly numeric requirements.
BibTeX
@misc{qi2026ask,
  author = {Jinhu Qi and Minda Hu and Wentao Zhang and Zixing Song and Irwin King},
  title = {{Ask Before You Answer: A Question-Driven Evidence Reasoning Architecture (QERA) for In-Context Learning}},
  year = {2026},
  eprint = {2610.35083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35083},
}
Figure