Discovering Chunks in Neural Embeddings for Interpretability

Shuchen Wu, Stephan Alaniz, Eric Schulz, Zeynep AkataarXiv 2025 · DOI · arXiv

Introduction

Neural networks are often described as "black boxes." While their inputs and outputs are visible, the internal dynamics of billions of parameters remain largely opaque. Current interpretability research typically focuses on either individual "monosemantic" neurons—which are rare—or low-dimensional projections that may discard critical information.

Researchers have proposed a new framework inspired by human cognition: chunking. In the human mind, high-dimensional sensory data is processed by grouping recurring patterns into meaningful entities, such as letters into words or words into phrases. This paper explores whether artificial neural networks (ANNs) naturally perform similar operations and if these "chunks" can be extracted to make models human-readable.

Methodology: The Reflection Hypothesis

The core of this research is the Reflection Hypothesis, which posits that a well-trained network’s internal computational trajectories should mirror the structured regularities of its training data. To extract these structures, the authors developed the Chunks from Neural Embeddings (CNE) framework.

The CNE framework maps raw population activity to discrete symbolic chunks, mirroring the way biological minds organize vocabulary and grammar.

The CNE framework employs three primary methods for chunk discovery:

  1. Discrete Sequence Chunking: In Recurrent Neural Networks (RNNs), individual neuron activations are clustered into discrete "symbols." Compression algorithms then identify recurring sequences of these symbols.
  2. Neural Population Averaging: For Large Language Models (LLMs), the mean activation vector (the "chunk") for a specific concept is identified. A "tolerance radius" is then used to detect when the model is processing that concept in real-time.
  3. Unsupervised Chunk Discovery: Using dictionary learning, the system identifies $K$ recurring chunks in the embedding space without prior knowledge of input labels. This is achieved by minimizing a loss function that encourages similarity between raw embeddings and the most representative chunk in a learned dictionary.
The RNN architecture used for sequence prediction, where hidden states $h$ are analyzed for recurring patterns.

Experimental Setup

The framework was tested across several domains:

  • Artificial Sequences: Simple "ABCD" patterns with varying noise levels were used for proof-of-concept in RNNs.
  • LLaMA3-8B: The primary LLM used to test the reflection hypothesis on natural language.
  • Project Gutenberg: Jane Austen’s Emma served as the corpus for unsupervised discovery and Part-of-Speech (POS) tagging correlation.
  • NLTK Penn Treebank: Used to evaluate if discovered chunks correspond to known linguistic structures.

Key Insights and Results

Validation of the Reflection Hypothesis

Neural activities are not random noise; they form trajectories that consistently reflect input data structure. In RNNs, the symbolic chunking method achieved perfect decoding accuracy, meaning internal chunks perfectly represented input regularities.

Causal Control via "Grafting"

The researchers demonstrated causal control by "grafting" hidden states. By manually setting a model's hidden state to a specific "chunk" vector, they could bias LLaMA3 to generate text about specific topics (e.g., "cheesecake") or accelerate learning in RNNs.

Temporal Specificity and Emergent Syntax

LLMs exhibit distinct neural signatures for a word depending on its temporal context: whether it is the current token, a past memory, or a future prediction. Furthermore, unsupervised chunks naturally aligned with POS tags. In early layers (1-10), some chunks showed a correlation of over 0.8 with specific grammatical markers, suggesting that models "chunk" syntax before semantics.

Limitations and Future Work

Current unsupervised methods are primarily limited to identifying single-token chunks. Capturing multi-token entities (e.g., "New York") remains a challenge. Additionally, population averaging requires prior knowledge of the target patterns.

Future research will focus on establishing stronger theoretical foundations for the Reflection Hypothesis and developing non-parametric methods for chunk discovery. The ultimate goal is a "universal dictionary" of neural chunks to render any high-dimensional model interpretable.

Summary updated

Abstract

Understanding neural networks is challenging due to their high-dimensional, interacting components. Inspired by human cognition, which processes complex sensory data by chunking it into recurring entities, we propose leveraging this principle to interpret artificial neural population activities. Biological and artificial intelligence share the challenge of learning from structured, naturalistic data, and we hypothesize that the cognitive mechanism of chunking can provide insights into artificial systems. We first demonstrate this concept in recurrent neural networks (RNNs) trained on artificial sequences with imposed regularities, observing that their hidden states reflect these patterns, which can be extracted as a dictionary of chunks that influence network responses. Extending this to large language models (LLMs) like LLaMA, we identify similar recurring embedding states corresponding to concepts in the input, with perturbations to these states activating or inhibiting the associated concepts. By exploring methods to extract dictionaries of identifiable chunks across neural embeddings of varying complexity, our findings introduce a new framework for interpreting neural networks, framing their population activity as structured reflections of the data they process.
BibTeX
@misc{wu2025discovering,
  author = {Shuchen Wu and Stephan Alaniz and Eric Schulz and Zeynep Akata},
  title = {{Discovering Chunks in Neural Embeddings for Interpretability}},
  year = {2025},
  eprint = {2502.01803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.01803},
}
Figure