Detecting the Semantic Fixed Point: A Geometric Framework for Efficient Inference

Jiawei Gu, Ziyue Qiao, Xiao LuoICML 2026 · OpenReview

TL;DR

This paper introduces a novel "Geometric Convergence Early Exit" (GCEE) method for Transformers that monitors the geometry of internal hidden states to determine when to stop computation. GCEE significantly reduces inference costs (30-35% FLOPs) while maintaining high accuracy (over 98%), outperforming existing confidence-based methods by offering a more reliable and computationally cheaper signal.

What Problem Does This Paper Address?

Large Language Models (LLMs) like Transformers are computationally expensive, requiring substantial resources for inference. While "early exit" strategies aim to reduce this cost by terminating computation before the final layer, existing methods often rely on output confidence. This approach has limitations: it can be misleading (a model can be confidently wrong) and, critically for LLMs, the computation of confidence involves projecting the hidden state through the language model head, which has a vocabulary-size dependent overhead ($O(d \times V)$). This overhead can negate the speedup gained from exiting early, especially with large vocabularies. The paper seeks a more efficient and reliable early exit criterion that directly monitors internal convergence without this vocabulary bottleneck.

Methods

The core idea is to treat Transformer layers as steps in an iterative refinement process, analogous to fixed-point iteration in numerical analysis. Instead of relying on output confidence, GCEE examines the geometry of the hidden state trajectory. The authors observe a consistent "two-phase structure" in layer-wise updates:

  1. Construction Phase: Early layers have large, volatile updates as the model builds its representation.
  2. Propagation Phase: Later layers exhibit small, stable, and aligned updates as the model propagates an already-formed representation.

This transition is sharp and input-dependent. GCEE formalizes this into a stopping criterion by tracking two geometric quantities at each layer:

  • Normalized Step Size ($\hat{s}^{(l)}$): The norm of the update vector ($u^{(l)} = h^{(l)} - h^{(l-1)}$) normalized by the hidden state norm ($||h^{(l)}||$). This measures how much the representation changes relative to its current scale.
  • Directional Stability ($c^{(l)}$): The cosine similarity between consecutive update vectors ($u^{(l)}$ and $u^{(l-1)}$). This measures how consistently the representation is evolving.

The model exits when both the step size becomes small and the direction stabilizes for a few consecutive layers, after a minimum warmup depth. This criterion has an $O(d)$ overhead per layer, independent of vocabulary size.

Datasets

The paper evaluates its method on several standard NLP benchmarks:

  • Question Answering: TriviaQA, Natural Questions
  • Commonsense Reasoning: HellaSwag
  • Coreference Resolution: WinoGrande

These datasets were used for both training/validation (for hyperparameter tuning) and for final evaluation.

Key Insights and Ideas

  • Geometric Convergence as a Reliable Signal: Output confidence is a symptom of convergence, not a direct measure. Geometric properties of internal states provide a more direct and robust signal for when a representation has stabilized.
  • The Two-Phase Structure is Universal: The observed "construction" to "propagation" phase transition in hidden state updates is consistent across different Transformer architectures (decoder-only, encoder-decoder), model scales, and tasks. This suggests it's a fundamental property of how Transformers learn.
  • Input-Dependent Saturation: The point at which computation can be safely halted (the "saturation layer") varies with input difficulty. Simpler inputs saturate earlier, while complex ones require deeper processing. GCEE naturally adapts to this.
  • Vocabulary-Free Overhead: By focusing on geometric properties of vectors ($O(d)$), GCEE avoids the costly vocabulary projection ($O(d \times V)$) required by confidence-based methods, making it significantly more efficient for LLMs.
  • "Semantic Fixed Point": The paper frames the stable state of the hidden representation as a "semantic fixed point," which the geometric criterion aims to detect.

Key Results

  • FLOPs Reduction: GCEE reduces FLOPs by 30-35% on average across LLaMA-2-7B and LLaMA-2-13B models.
  • Accuracy Retention: Over 97% of full-depth accuracy is retained.
  • Speedup: Achieves a 1.45x end-to-end speedup, significantly outperforming entropy-based methods (1.14x) whose gains are eroded by vocabulary overhead.
  • Robustness: The criterion is insensitive to small threshold variations, simplifying deployment.
  • Error Attribution: GCEE rarely causes errors due to premature exits (only 18% of errors attributed to this); most errors stem from inherent model limitations.

Limitations and Future Work

  • Hyperparameter Tuning: While GCEE is generally robust, optimal thresholds ($T_s, T_c, l_{min}, K$) might still benefit from task-specific tuning. The authors propose a "self-calibrating" variant that achieves comparable performance with minimal tuning.
  • Theoretical Guarantees: The paper provides theoretical analysis and bounds on logit perturbation and decision invariance, but further work could explore deeper theoretical connections between geometric convergence and semantic understanding.
  • Generalization to Other Architectures: While tested on various Transformer variants, future work could explore its applicability to non-Transformer architectures or multimodal models.
  • Malicious Use: The paper acknowledges that efficient inference could lower the cost of malicious applications, but states their method doesn't introduce new capabilities that inherently increase safety risks.

Summary updated

Abstract

Each layer of a Transformer refines the hidden state toward a prediction, an iterative process resembling fixed-point iteration. Yet when should this iteration terminate? Existing early exit methods rely on output confidence as a proxy for internal convergence. We take a more direct approach by examining the geometry of the hidden state trajectory. We find that layer-wise updates exhibit a two-phase structure: large, volatile updates in early layers, followed by small, aligned updates as the model propagates an already-formed representation. The transition is remarkably sharp. This yields a simple criterion: exit when step size vanishes and direction stabilizes. We track the normalized update norm and cosine similarity between consecutive updates, exiting when both indicate convergence. The overhead is $O(d)$ per layer, independent of vocabulary size, requiring no learned components or architectural modifications. On LLaMA-2-7B and LLaMA-2-13B across question answering and commonsense reasoning tasks, this geometric criterion reduces FLOPs by 30--35\% while retaining over 98\% of full-depth accuracy.
BibTeX
@inproceedings{gu2026detecting,
  author = {Jiawei Gu and Ziyue Qiao and Xiao Luo},
  title = {{Detecting the Semantic Fixed Point: A Geometric Framework for Efficient Inference}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=DACN5xM4h7},
}
Figure