Introduction
The rapid expansion of sequence model architectures—from standard Transformers to state-space models (SSMs) like Mamba and various linear attention mechanisms—has created a need for better comparative metrics. Traditionally, researchers have relied on theoretical memory capacity (e.g., state dimension) or qualitative visualizations. However, theoretical capacity often fails to reflect actual performance because models may not effectively utilize their available state. This paper introduces Effective State-Size (ESS), a novel metric derived from control theory that quantifies actual memory utilization, providing a more accurate predictor of performance on recall-heavy tasks.
The Effective State-Size Framework
To create a unified metric, the authors model sequence architectures as Linear Input-Varying (LIV) systems. In this view, computations like softmax attention, convolutions, and recurrences are expressed as linear operators ($y = Tu$) mapping past inputs to future outputs. ESS is defined as the rank of the submatrix of $T$ that represents the causal flow of information from the past to the future, drawing on minimal realization theory from signal processing.
Because exact rank computation is numerically unstable, two practical SVD-based variants are proposed: Tolerance-ESS, which counts singular values above a threshold, and Entropy-ESS, which uses normalized spectral entropy to provide a continuous, threshold-free metric.
Empirical Validation and Results
The utility of ESS was tested across a variety of synthetic tasks (e.g., Multi-Query Associative Recall) and language modeling datasets like FineWeb. The results demonstrate that theoretical capacity (Theoretically realizable State-Size, or TSS) is a poor proxy for performance. In contrast, ESS shows a remarkably strong correlation with task accuracy across different architectures.
As shown in the figure above, models like Softmax Attention (SA) and Gated Linear Attention (GLA) exhibit a clear phase transition where higher ESS directly translates to higher accuracy. Conversely, standard Linear Attention (LA) often struggles with "state collapse," where it fails to utilize its memory capacity effectively, leading to poor performance on complex recall tasks.
Practical Applications and Insights
Beyond benchmarking, ESS serves as a powerful diagnostic tool:
- Diagnosing Training Failures: ESS distinguishes between state saturation (capacity is full but insufficient) and state collapse (capacity is available but unused). This distinction guides whether to increase model size or improve initialization/regularization.
- Architecture Optimization: Analysis of ESS at initialization revealed a scaling flaw in the Mamba (S6) featurizer. By adjusting initialization based on ESS feedback, the authors were able to fix the model's ability to learn associative recall tasks.
- Model Distillation: In compression tasks, a teacher model's ESS predicts distillation loss. Layers with high ESS are denser in information and harder to compress without significant performance drops.
- Dynamic State Modulation: High-performing LLMs dynamically adjust their ESS based on context. For instance, ESS typically dips at end-of-speech tokens, indicating a quantitative "reset" of the model's working memory.
Conclusion
Effective State-Size provides a rigorous, quantitative bridge between the theoretical design of sequence models and their empirical performance. By shifting the focus from capacity to utilization, ESS enables more principled architecture design, better diagnostic capabilities during training, and deeper insights into how modern language models manage information over time. Future work may involve using ESS to optimize layer-mixing in hybrid models and investigating the strange ESS dynamics observed during the early stages of training.
Summary updated