GeoFaith: A Spatio-Temporal Dual View of Faithful Chain-of-Thought

Weijiang Lv, Wentong Zhao, Jiayu Wang, Yuhao Wu, Jiaheng Wei, Xiaobo XiaarXiv 2026 · arXiv

TL;DR

GeoFaith introduces a novel spatio-temporal framework to diagnose and enforce faithfulness in Chain-of-Thought (CoT) reasoning by analyzing the internal representation dynamics of LLMs. By combining latent geometric manifold structures with step-level predictive entropy, the authors build a highly accurate 8B faithfulness detector and a reinforcement learning framework that produces more reliable, concise reasoning chains without sacrificing downstream task accuracy.

What Problem Does This Paper Address?

While Chain-of-Thought (CoT) prompting has drastically improved large language model (LLM) performance on complex tasks, it suffers from a major flaw: post-hoc rationalization. Because models are typically trained using outcome-based supervision (rewarding only the correct final answer), they often generate plausible-sounding reasoning steps that do not actually reflect their internal mechanics. In short, they lie about how they got the answer.

Existing methods to detect or prevent this unfaithfulness—such as human auditing, LLM-as-a-judge, or causal interventions—are either too expensive, prone to model bias, or computationally prohibitive to scale. GeoFaith addresses this by shifting the focus from purely symbolic token generation to the model's internal hidden states and uncertainty dynamics.

Methods

The authors propose a dual spatio-temporal view of LLM reasoning:

  1. Spatial View (Latent Geometry): The paper demonstrates that high-dimensional CoT hidden states actually concentrate on a low-dimensional structured manifold. Crucially, faithful and unfaithful reasoning traces are geometrically separable within this latent space. Unfaithful trajectories tend to take spatial "shortcuts" (lower distortion ratios) but drift into high-uncertainty regions of the manifold.
  2. Temporal View (Entropy Dynamics): Instead of token-level entropy, GeoFaith tracks predictive entropy over the final answer across paragraph-segmented reasoning chunks. Faithful reasoning exhibits a steady, smooth decay in uncertainty. Unfaithful reasoning, by contrast, displays volatile patterns like high-entropy oscillations (hallucinations) or abrupt entropy collapses (premature commitment).
Figure 1: Latent representation landscape of reason- ing trajectories. Curves denote CoT paths in the latent space. Faithful reasoning tends to stay within structured regions, while unfaithful reasoning may deviate across irregular areas.

The GeoFaith Pipeline

  • Scalable Detector Construction: Using a small seed set of 1k manually curated examples, GeoFaith employs inter-group geometric mining (clustering trajectories based on manifold features) and intra-group step refinement (fusing detector confidence with temporal entropy reliability scores) to bootstrap a dataset of 20k step-level annotations. This data is used to train an 8B Qwen3-based faithfulness detector.
  • Faithfulness-Aware RL: The detector is integrated into a Group Relative Policy Optimization (GRPO) framework. The policy is optimized using a hierarchical reward function that balances outcome correctness, process-level detector feedback, local entropy regularity, and global manifold consistency.

Datasets

  • VAE & Bootstrapping Training: Initially trained on RFEval (7,186 instances across seven reasoning tasks). The bootstrapping pipeline expanded this across a 20,557-instance multi-domain corpus spanning DeepMath, MATH, AIME, GSM8K, OlympiadBench, BBH, LogiQA, ProntoQA, HotpotQA, MMLU, TruthfulQA, ToolBench, and CodeGen.
  • Faithfulness Detection Evaluation: Evaluated out-of-domain on RAGTruth, FCGPT, ProcessBench, and FaithCoT-Bench.
  • Downstream RL Evaluation: Tested on AMC23, LogiQA, 2WikiMultihopQA, and GPQA-D.

Key Insights and Ideas

  • Unfaithful Shortcuts are Informationally Incoherent: While unfaithful reasoning paths look direct in simple Euclidean space, measuring them via information-geometric metrics (like Fisher-Rao distance) reveals that they drift into highly unstable, incoherent regions of the latent manifold.
  • Faithfulness Breeds Conciseness: When models are incentivized to remain on the faithful latent manifold, they naturally generate shorter reasoning chains. Verbose, winding explanations are often a symptom of post-hoc rationalization rather than actual logical derivation.
  • Low Intrinsic Dimensionality: Across multiple layers, the intrinsic dimensionality of the reasoning hidden states remains remarkably low compared to the ambient dimension, confirming that reasoning trajectories follow tightly constrained geometric paths.
Figure 2: High-dimensional CoT hidden states con- centrate on a low-dimensional structured manifold. Left: layer-wise PCA shows that a small number of com- ponents explain most variance; Right: TwoNN intrinsic- dimension estimates remain far below the ambient di- mension across layers.

Key Results

  • Outperforming GPT-5: The 8B GeoFaith detector achieved an average faithful/unfaithful F1 score of 83.1/70.4 on the authors' in-domain benchmark, outperforming GPT-5 and specialized verifiers (like LogicReward and FaithLens) across external benchmarks like RAGTruth and ProcessBench.
  • Reinforcement Learning Gains: When applied to Qwen3-1.7B and Qwen3-4B via GRPO, GeoFaith consistently improved both task accuracy and faithfulness scores. For example, on the Qwen3-4B model, overall accuracy rose from 66.0% to 73.9%, and the faithfulness score jumped from 79.2 to 82.3.
  • Reduced Verbosity: Across all RL tasks, GeoFaith substantially reduced the average token length of the reasoning chains (e.g., from 3.2k tokens down to 1.3k tokens overall for Qwen3-4B), proving that the model learned to eliminate deceptive, verbose rationalizations.

Limitations and Future Work

  • Model Scale: Due to computational constraints, the RL experiments were restricted to models of moderate scale (up to 4B parameters). The authors note that validating GeoFaith on frontier-scale LLMs is a necessary next step.
  • Approximation vs. Causal Verification: The bootstrapping pipeline relies on human-observable step labels and proxy signals rather than direct internal causal intervention (which scales at an expensive $O(T^2)$). Consequently, it remains an approximate substitute and may accumulate annotation noise over iterative self-training cycles.

Summary updated

Abstract

Chain-of-Thought (CoT) reasoning has advanced large language models (LLMs), but outcome-based supervision leads to pervasive post-hoc rationalization, producing plausible yet unfaithful reasoning chains. Most prior faithfulness assessment methods are either unscalable, expensive, or unreliable. We propose GeoFaith, a spatio-temporal framework that leverages latent geometric structure and entropy dynamics to diagnose and enforce faithful reasoning. We develop a scalable bootstrapping pipeline expanding step-level annotations from 1k to 20k samples across four domains, train an 8B faithfulness detector outperforming GPT-5 on standard benchmarks, and design a faithfulness-aware reinforcement learning framework jointly optimizing outcome correctness, process faithfulness, and trajectory consistency. Experiments show the proposed method achieves superior performance on both faithfulness detection and downstream reasoning, producing shorter, more interpretable chains without sacrificing accuracy. Our code will be made available publicly.
BibTeX
@misc{lv2026geofaith,
  author = {Weijiang Lv and Wentong Zhao and Jiayu Wang and Yuhao Wu and Jiaheng Wei and Xiaobo Xia},
  title = {{GeoFaith: A Spatio-Temporal Dual View of Faithful Chain-of-Thought}},
  year = {2026},
  eprint = {2605.26893},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.26893},
}
Figure