Higher-Order Evidence & the Limits of Self-Correction
TL;DR
This paper challenges the common assumption that AI agents should always align their beliefs with their own self-assessments. By demonstrating that an agent’s internal signals can be more accurate than its explicit self-critiques, the author argues that forcing "coherence" through a veto or training for agreement can actually degrade an agent's performance and destroy its epistemic competence.
What Problem Does This Paper Address?
In the quest for safer and more reliable AI, researchers often build agents that can "self-correct." This usually involves a two-part system: a process that generates a conclusion and a "self-assessment" process that evaluates whether that conclusion is justified. If the evaluator flags a mistake, the standard design choice is to have the agent abandon the conclusion—a principle the paper calls Self-Assessment Authority.
The author addresses a phenomenon called epistemic akrasia: a state where an agent believes something while simultaneously judging that it isn't justified in doing so. In humans, this is often seen as a rational failure. In AI, we assume that resolving this conflict (making the agent "coherent") is always an improvement. This paper asks: Is that assumption actually true? Or could an agent be "right to be wrong" about its own reasoning?
Methods
The paper is a theoretical and philosophical analysis grounded in formal epistemology and decision theory. It introduces several key conceptual frameworks:
- Dissociation: The author distinguishes between the information a system contains (internal states), the information it uses to keep a conclusion (internal signals like logit-based confidence), and the information available to its explicit self-assessment (what it "says" about its reasoning).
- Decision-Theoretic Modeling: In Section 4, the author uses a worked example with 1,000 cases to quantify the "cost" of different policies (Deference vs. Signal-guided retention).
- Comparative Authority: A proposed alternative to the "veto" rule. Instead of the evaluator having the power to delete a belief, it has the authority to trigger further scrutiny (verification) without necessarily forcing the agent to abandon its initial conclusion.
Datasets
Not discussed in the paper. As a theoretical work, it does not introduce new datasets. However, it cites and analyzes results from several recent empirical studies on Large Language Models (LLMs), including:
- Orgad et al. (2025): On internal classifiers detecting errors better than the model’s own verbal reports.
- Kumaran et al. (2025/2026): On how internal confidence signals drive model behavior and resistance to advice.
- Huang et al. (2024): On how "self-correction" without external feedback often leads models to change correct answers to incorrect ones.
Key Insights and Ideas
- The "Veto" Can Be Harmful: If an agent has an internal signal that tracks the "soundness" of its reasoning better than its explicit evaluator does, forcing the agent to defer to the evaluator (the "veto") will cause it to discard correct conclusions.
- Agreement $\neq$ Improvement: You can achieve "agreement" between a model and its evaluator in two ways: by fixing the model or by degrading the evaluator into a mere "echo" that approves whatever the model says. Training for agreement alone cannot distinguish between these two outcomes.
- The Disagreement Selection Bias: A self-assessment might be 90% reliable overall, but it can be wrong 100% of the time in the specific cases where it disagrees with the agent's internal signals. General reliability is no guide for the cases where the veto actually intervenes.
- Akrasia as Competence: An agent that remains "akratic"—keeping a belief its evaluator dislikes—might actually be exercising a higher level of epistemic competence by relying on superior, non-verbalized information.
Key Results
The paper’s core quantitative argument is found in its decision-theoretic example:
- Scenario: 1,000 cases where an evaluator is 90% accurate.
- Policy A (Deference/Veto): The agent gives up every conclusion the evaluator flags. Result: Score 0 (it avoids all bad conclusions but loses all good ones).
- Policy B (Signal-guided retention): The agent uses an internal signal (dissociated from the evaluator) to decide which flagged conclusions to keep. Result: Score +50.
- Conclusion: The "akratic" agent (Policy B) significantly outperforms the "coherent" agent (Policy A). Forcing the agent to resolve its internal conflict would result in a net loss of 50 units of "truth."
Limitations and Future Work
- Soundness vs. Correctness: Most current research tracks whether a model's answer is correct, but the author notes that epistemic justification depends on whether the reasoning was sound. Future empirical work needs to find signals that specifically track reasoning quality, not just output accuracy.
- The "Possession" Question: Philosophers may argue over whether an internal signal the agent can't "talk about" counts as evidence. The author suggests this is a design choice for AI, but a deep philosophical hurdle for human-like attribution of beliefs.
- Experimental Testing: The paper concludes by proposing "Double Dissociation" experiments—manipulating internal signals and evaluator records independently to see if agents exhibit the predicted patterns of behavior. This remains to be executed in a lab setting.
Summary updated