Efficient Latent Semantic Clustering for Scaling Test-Time Computation of LLMs

Sung Jae Lee, Hoyoung Kim, Jeongyeon Hwang, Eunhyeok Park, Jungseul Ok2025 · DOI

Introduction

To enhance the reliability of Large Language Models (LLMs), researchers increasingly utilize "test-time scaling" techniques. These methods involve generating multiple responses to a single prompt to either quantify uncertainty or explore diverse reasoning paths, such as in "Tree-of-Thoughts" frameworks. However, a significant bottleneck in this process is semantic clustering: the necessity of grouping semantically equivalent outputs to avoid redundant computation and accurately calculate confidence scores.

Traditional approaches rely on external "judge" models—such as Natural Language Inference (NLI) or embedding models—to compare pairs of generated sequences. These external models are computationally expensive, require substantial VRAM, and often lack the original context of the prompt, leading to errors in semantic grouping. This paper introduces Latent Semantic Clustering (LSC), a lightweight method that leverages the generator LLM's internal hidden states to achieve superior clustering performance with near-zero additional memory and a massive reduction in latency.

The LSC Methodology

LSC operates on the principle that an LLM’s internal representations are inherently context-aware. Unlike "black-box" methods that treat the LLM as a text generator and use a second model for analysis, LSC treats the LLM as a "white-box." The process consists of three primary stages:

  1. Hidden State Extraction: During the generation of $N$ sequences, the method captures the hidden state vector from an intermediate Transformer layer at the final token of each sequence.
  2. Adjacency Matrix Construction: The system calculates the pairwise cosine similarity between these hidden vectors, creating a map of how the LLM perceives the similarity of its own outputs.
  3. Spectral Clustering: Using the eigenvalues of the similarity matrix, the method automatically determines the optimal number of clusters ($k$) and groups the sequences accordingly.

By utilizing the internal "thoughts" of the model, LSC ensures that the clustering is conditioned on the initial prompt, capturing nuances that external models might miss.

Experimental Evaluation

The researchers evaluated LSC across two primary domains using models including Llama3-8B-Instruct, Mistral-7B-Instruct, and Llama3.2-1B:

  • Uncertainty Quantification: Tested on BioASQ (biomedical QA), SQuAD (Wikipedia QA), and TriviaQA.
  • Multi-step Reasoning: Tested on GSM8K (math) and ARC (science questions).

Key Insights

  • Contextual Superiority: External models often fail because they only see the generated answers. For example, in a specific context, "2 a night" and "60 a month" are semantically identical. LSC captures this effortlessly because its internal states are already conditioned on the prompt.
  • Layer Dynamics: The study found that intermediate and later layers provide more robust semantic representations than early layers, which focus more on low-level token features.
  • Zero-Cost Efficiency: Since hidden states are computed during generation, LSC adds virtually no computational overhead.

Results and Impact

LSC demonstrates significant improvements over traditional baselines in three critical areas:

  1. Speed and Memory: LSC reduced clustering latency from seconds to milliseconds (e.g., 0.0069s compared to 4.91s for a Mistral-7B baseline). It requires approximately 0 GB of additional memory, whereas external models can demand up to 20 GB.
  2. Clustering Quality: LSC achieved an F1 score of 0.8789, significantly outperforming NLI-based (0.6904) and Sentence-BERT (0.7198) baselines.
  3. Reasoning and Uncertainty: When integrated into reasoning frameworks (SExp-LSC), it reduced the number of required LLM inferences by 22.58% on the ARC dataset while maintaining or improving accuracy. Furthermore, LSC consistently achieved the highest AUROC scores for uncertainty quantification, indicating superior predictive power regarding model errors.

Limitations and Future Work

While LSC is highly efficient, it requires access to the model's internal hidden states, making it incompatible with closed, API-only models like GPT-4 unless latent vectors are exposed. Future research may explore aggregating information across multiple layers or tokens to further refine performance and investigate methods to automate the tuning of the cluster-determination threshold ($\tau$).

Conclusion

Latent Semantic Clustering represents a shift toward more efficient test-time computation. By looking inside the "brain" of the LLM, LSC provides a faster, more accurate, and more contextually aware method for semantic grouping, paving the way for more scalable and reliable AI reasoning.

Summary updated

BibTeX
@misc{lee2025efficient,
  author = {Sung Jae Lee and Hoyoung Kim and Jeongyeon Hwang and Eunhyeok Park and Jungseul Ok},
  title = {{Efficient Latent Semantic Clustering for Scaling Test-Time Computation of LLMs}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.1310},
}
Figure