Beyond Answers, Traces, and Efficiency: Evidence for Adaptive Reasoning

Sejin Kim, Sundong KimarXiv 2026 · arXiv

TL;DR

This paper argues that current AI reasoning benchmarks—which focus on final answers, reasoning traces, and efficiency—fail to prove that a system is actually reasoning adaptively. The authors propose a new "intervention-oriented" evaluation framework that tests how a model's behavior changes when task structures or evidence are deliberately altered, providing a more rigorous way to measure abstraction transfer and hypothesis revision.

What Problem Does This Paper Address?

We are currently in an era of "System-2" LLMs that use test-time compute (like search and self-correction) to solve complex problems. However, our evaluation metrics haven't kept up.

The authors point out a critical gap: Observation is not intervention. * Accuracy doesn't tell you if a model used a transferable abstraction or just got lucky with a memorized pattern. * Reasoning traces (CoT) might be "unfaithful"—the model might provide a plausible explanation that doesn't actually drive its decision. * Efficiency metrics don't reveal if a model learned something reusable or just searched a larger space.

Without testing how a system responds to change, we cannot distinguish between a robust reasoner and a brittle system that succeeds through massive sampling or narrow heuristics.

Figure 1: Existing evidence characterizes what was observed; intervention-oriented evaluation tests how behavior changes under controlled changes to tasks or evidence.

Methods

The authors propose Intervention-Oriented Evaluation, characterized by two complementary scales of adaptivity:

  1. Across-task adaptivity: Tests if acquired abstractions transfer. This involves creating paired task variants:
  2. Concept-preserving interventions: Change surface features (colors, layout) while keeping the core logic the same. Behavior should remain stable.
  3. Concept-changing interventions: Modify the core logic. Behavior should change predictably.
  4. Within-task adaptivity: Tests if a model revises its hypothesis when faced with new evidence. This uses a "same-history branching" protocol where the evaluator introduces a signal (feedback or a counterexample) and checks if the model's subsequent intermediate decisions ($Z_t$) and behavior ($B_{>t}$) update appropriately.

Crucially, the framework requires explicit reporting of test-time budgets (samples, tokens, search steps). This ensures that improvements are attributed to better reasoning rather than simply "more rolls of the dice."

Datasets

The paper uses the ARC-AGI (Abstraction and Reasoning Corpus) as a diagnostic case study:

  • ARC-AGI-1 & 2: Used to demonstrate across-task adaptivity through static input-output transformations.
  • ARC-AGI-3: An interactive environment used to demonstrate within-task adaptivity, where an agent must act, observe consequences, and revise its strategy.

The authors also reference ConceptARC, a benchmark that groups ARC tasks by human-interpretable concepts, as a foundation for constructing their proposed interventions.

Key Insights and Ideas

  • The "Consistency" Requirement: For across-task adaptivity, the authors define the relation $\Delta C \sim \Delta B$. This means the change in behavior ($\Delta B$) must be systematically consistent with the controlled change in task structure ($\Delta C$).
  • Revision vs. Repeated Search: A major "aha moment" is the distinction between a model that revises a failed hypothesis and one that just retries until it hits the right answer. Within-task evaluation aims to isolate the former by branching from the exact same pre-intervention state.
  • Selective Adaptation: Adaptivity isn't just about changing behavior; it's about selective change. A system that changes its mind due to irrelevant noise is unstable, not adaptive. Robust reasoning requires stability toward irrelevant changes and sensitivity toward relevant ones.
  • Alignment Building Blocks: While adaptivity isn't the same as alignment, it provides the behavioral evidence needed for alignment-relevant properties like corrigibility (responding to human correction) and auditability (making reasoning steps falsifiable).
Figure 2: The two scales of adaptive reasoning: across-task transfer and within-task revision.

Key Results

As a framework paper, the "results" are qualitative worked examples that demonstrate how to apply these protocols to ARC tasks:

  • Across-task Example: In Figure 3, the authors show how a "pixel-level downward relocation" rule can be tested. A concept-preserving variant adds a background (the rule stays the same), while a concept-changing variant moves "objects" instead of pixels. An adaptive reasoner should succeed at the first and change its strategy for the second.
  • Within-task Example: In Figure 4, an agent in an interactive ARC-AGI-3 task holds a hypothesis about how endpoints control an object. When a "role-swap" event occurs (contradictory evidence), the evaluator checks if the agent's next "intermediate decision" (its internal plan or proposal) actually reflects this new information.

The paper provides a Minimum Reporting Checklist (Table 2) to standardize how researchers should present evidence for adaptive reasoning, moving beyond simple accuracy scores.

Limitations and Future Work

  • No Internal Mechanism: The authors admit this is a "black-box" behavioral framework. It proves that a system adapts, but not how it does so internally.
  • Domain Specificity: Designing valid interventions (especially concept-changing ones) requires deep domain knowledge. There is no "universal template" for interventions yet.
  • Measurement Uncertainty: Observable proxies (like a model's written plan) might only partially reflect the actual decision process.
  • Future Direction: The authors suggest applying this logic to other domains like program synthesis, tool-use agents, and complex planning tasks where "System-2" behavior is prevalent.

Summary updated

Abstract

Current AI systems are evaluated as reasoners through final-answer accuracy, reasoning traces, process supervision, and computational or human-relative efficiency. Yet these measures do not establish whether reasoning adapts when tasks or evidence change. We distinguish across-task adaptivity, where acquired abstractions transfer or recombine across novel problems, fromwithin-task adaptivity, where hypotheses and actions are revised in response to new observations, feedback, or counterexamples. Using ARC-AGI as a diagnostic case study rather than an alignment benchmark, we examine what existing measures leave unresolved when test-time search, sampling, or adaptation affects performance. We propose intervention-oriented evaluation that tests abstraction transfer and hypothesis revision under controlled changes to tasks and evidence, with explicit reporting of test-time procedures and budgets. This perspective supports evaluating systems for robustness, responsiveness to corrective feedback, and auditability under changing conditions.
BibTeX
@misc{kim2026beyond,
  author = {Sejin Kim and Sundong Kim},
  title = {{Beyond Answers, Traces, and Efficiency: Evidence for Adaptive Reasoning}},
  year = {2026},
  eprint = {2611.02704},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02704},
}
Figure