Quantifying Memory Utilization with Effective State-Size

Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas, Alessandro Moro, Qi An, Taiji Suzuki, Atsushi Yamashita, Michael Poli, Stefano Massaroli

Introduction

The rapid expansion of sequence model architectures—from standard Transformers to state-space models (SSMs) like Mamba and various linear attention mechanisms—has created a need for better comparative metrics. Traditionally, researchers have relied on theoretical memory capacity (e.g., state dimension) or qualitative visualizations. However, theoretical capacity often fails to reflect actual performance because models may not effectively utilize their available state. This paper introduces Effective State-Size (ESS), a novel metric derived from control theory that quantifies actual memory utilization, providing a more accurate predictor of performance on recall-heavy tasks.

The Effective State-Size Framework

To create a unified metric, the authors model sequence architectures as Linear Input-Varying (LIV) systems. In this view, computations like softmax attention, convolutions, and recurrences are expressed as linear operators ($y = Tu$) mapping past inputs to future outputs. ESS is defined as the rank of the submatrix of $T$ that represents the causal flow of information from the past to the future, drawing on minimal realization theory from signal processing.

Overview of the ESS framework, mapping causal sequence models to quantifiable memory utilization metrics.

Because exact rank computation is numerically unstable, two practical SVD-based variants are proposed: Tolerance-ESS, which counts singular values above a threshold, and Entropy-ESS, which uses normalized spectral entropy to provide a continuous, threshold-free metric.

Empirical Validation and Results

The utility of ESS was tested across a variety of synthetic tasks (e.g., Multi-Query Associative Recall) and language modeling datasets like FineWeb. The results demonstrate that theoretical capacity (Theoretically realizable State-Size, or TSS) is a poor proxy for performance. In contrast, ESS shows a remarkably strong correlation with task accuracy across different architectures.

Correlation between normalized Effective State-Size (ESS/kv) and task accuracy across various architectures.

As shown in the figure above, models like Softmax Attention (SA) and Gated Linear Attention (GLA) exhibit a clear phase transition where higher ESS directly translates to higher accuracy. Conversely, standard Linear Attention (LA) often struggles with "state collapse," where it fails to utilize its memory capacity effectively, leading to poor performance on complex recall tasks.

Practical Applications and Insights

Beyond benchmarking, ESS serves as a powerful diagnostic tool:

  • Diagnosing Training Failures: ESS distinguishes between state saturation (capacity is full but insufficient) and state collapse (capacity is available but unused). This distinction guides whether to increase model size or improve initialization/regularization.
  • Architecture Optimization: Analysis of ESS at initialization revealed a scaling flaw in the Mamba (S6) featurizer. By adjusting initialization based on ESS feedback, the authors were able to fix the model's ability to learn associative recall tasks.
  • Model Distillation: In compression tasks, a teacher model's ESS predicts distillation loss. Layers with high ESS are denser in information and harder to compress without significant performance drops.
  • Dynamic State Modulation: High-performing LLMs dynamically adjust their ESS based on context. For instance, ESS typically dips at end-of-speech tokens, indicating a quantitative "reset" of the model's working memory.

Conclusion

Effective State-Size provides a rigorous, quantitative bridge between the theoretical design of sequence models and their empirical performance. By shifting the focus from capacity to utilization, ESS enables more principled architecture design, better diagnostic capabilities during training, and deeper insights into how modern language models manage information over time. Future work may involve using ESS to optimize layer-mixing in hybrid models and investigating the strange ESS dynamics observed during the early stages of training.

Summary updated

BibTeX
@inproceedings{parnichkun2025quantifying,
  author = {Rom N. Parnichkun and Neehal Tumma and Armin W. Thomas and Alessandro Moro and Qi An and Taiji Suzuki and Atsushi Yamashita and Michael Poli and Stefano Massaroli},
  title = {{Quantifying Memory Utilization with Effective State-Size}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=YKAnIvY5hf},
}
Figure