A Hopfieldian View-based Interpretation for Chain-of-Thought Reasoning

Lijie Hu, Liang Liu, Shu Yang, Xin Chen, Hongru Xiao, Mengdi Li, Pan Zhou, Muhammad Asif Ali, Di WangarXiv 2024 · OpenReview

Abstract

Chain-of-Thought (CoT) reasoning is a pivotal technique for enhancing the performance of Large Language Models (LLMs) on complex tasks. However, the internal mechanisms that allow simple prompts like "let’s think step by step" to unlock these capabilities remain poorly understood. This paper introduces an explainable framework grounded in the "Hopfieldian view" from cognitive neuroscience, which treats cognition as transformations within representational spaces. We propose a "Read-and-Control" approach that treats prompts as stimuli to activate latent concepts. Our framework enables the localization of reasoning errors and the mathematical steering of models toward correct reasoning paths, demonstrating significant accuracy improvements across multiple reasoning benchmarks.

Introduction: The Mystery of Chain-of-Thought

While Chain-of-Thought (CoT) prompting has dramatically improved LLM performance, the field lacks a fundamental, top-down explanation for how these prompts alter a model's internal state. Most research focuses on empirical trial-and-error or external knowledge retrieval. This paper addresses two core mysteries:

  1. Zero-shot CoT: Why does a simple instruction like "let’s think step by step" fundamentally change the model's output?
  2. Few-shot CoT: Why do a few preceding examples drastically improve the model's logical consistency?

The Hopfieldian Framework and Read-and-Control

The authors build their framework on the Hopfieldian view, shifting the focus from individual neurons to the movement of representations through high-dimensional space. Their "Read-and-Control" approach treats LLM prompts as biological-style stimuli that shift the model's internal geometry into a more logical "action space."

The Read-and-Control framework based on the Hopfieldian view.

The framework consists of three main components:

  • Concept Modeling & Simulation: Identifying latent concepts (e.g., "careful reasoning") learned during pre-training that are activated by specific stimuli.
  • Representations Reading: Using Linear Artificial Tomography (LAT) and Principal Component Analysis (PCA) to extract a "reading vector" that represents the direction of a specific concept within the model's hidden layers.
  • Representations Controlling: Once a reading vector is identified, it can be mathematically injected back into the model's hidden states during inference. This "nudges" the model toward a more accurate reasoning path without needing to change the input prompt.

Key Results and Interpretability

The framework was evaluated across seven datasets covering arithmetic (GSM8K, SVAMP, AQuA), commonsense (StrategyQA), and symbolic reasoning. The "Read-and-Control" approach consistently outperformed standard baselines on models like Mistral-7B and LLaMA-3-8B, with accuracy gains of up to 3.81% on AQuA.

Reasoning Error Localization

A major contribution of this work is the ability to pinpoint exactly where a model’s reasoning goes wrong. By comparing the model's hidden states against the reading vector, the framework generates a "salience map."

Reasoning error localization visualized through a salience map.

As shown in the visualization, green and red bars indicate which parts of a reasoning chain are logically sound or flawed. This allows for "rectifying" reasoning paths—for instance, correcting a failed comparison between the depth of Mount Fuji and the Sea of Japan by applying the control vector.

Overcoming the "Stereotype" Trap

The authors discovered that few-shot examples can create "stereotypes," where the model forces a problem into a learned pattern even if the facts do not fit. The "Control" method proved capable of breaking these biases, demonstrating that the underlying logic is often already present in the model but requires precise activation.

Conclusion

By bridging neuroscience and AI, this paper provides a new lens for understanding LLM behavior. The results suggest that reasoning capabilities are latent within pre-trained models; CoT prompts are merely one way to access them. The "Read-and-Control" framework offers a more direct, mathematical method to steer these models toward logical accuracy. Future work will explore extending this approach to multi-modal data and automating the discovery of concept stimuli.

Summary updated

Abstract

Chain-of-Thought (CoT) holds a significant place in augmenting the reasoning performance for large language models (LLMs). While some studies focus on improving CoT accuracy through methods like retrieval enhancement, yet a rigorous explanation for why CoT achieves such success remains unclear. In this paper, we analyze CoT methods under two different settings by asking the following questions: (1) For zero-shot CoT, why does prompting the model with "let's think step by step" significantly impact its outputs? (2) For few-shot CoT, why does providing examples before questioning the model could substantially improve its reasoning ability? To answer these questions, we conduct a top-down explainable analysis from the Hopfieldian view and propose a Read-and-Control approach for controlling the accuracy of CoT. Through extensive experiments on seven datasets for three different tasks, we demonstrate that our framework can decipher the inner workings of CoT, provide reasoning error localization, and control to come up with the correct reasoning path.
BibTeX
@misc{hu2024hopfieldian,
  author = {Lijie Hu and Liang Liu and Shu Yang and Xin Chen and Hongru Xiao and Mengdi Li and Pan Zhou and Muhammad Asif Ali and Di Wang},
  title = {{A Hopfieldian View-based Interpretation for Chain-of-Thought Reasoning}},
  year = {2024},
  eprint = {2406.12255},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.12255},
}
Figure