Scaling Laws for SAE Training Data

Nikita Koriagin, Nikita Balagansky, Daniil GavrilovarXiv 2026 · arXiv

TL;DR

Training Sparse Autoencoders (SAEs) for mechanistic interpretability is notoriously storage-heavy, often requiring terabytes of saved activations. This paper demonstrates that SAE performance plateaus after a certain threshold of unique data; beyond this point, replaying a fixed buffer is just as effective as using fresh samples, allowing for 8–64× storage reductions with negligible loss in reconstruction or interpretability.

What Problem Does This Paper Address?

Sparse Autoencoders (SAEs) are the current "gold standard" for mechanistic interpretability, helping researchers decompose complex neural representations into human-understandable features. However, training them at scale is a logistical nightmare.

To train an SAE effectively, you typically need to present it with billions of activations from a base model. Storing these activations—rather than re-generating them on the fly for every different SAE size or architecture you want to test—is a common bottleneck. If every training update requires a fresh activation, storage requirements can easily reach several terabytes. The authors ask: How many unique activations do we actually need to store before we can just start replaying them without hurting the SAE's quality?

Methods

The researchers systematically decoupled two variables:

  1. Total presentations ($N_{total}$): The total number of activations shown to the optimizer (optimization effort).
  2. Unique activations ($U$): The number of distinct activations stored in the buffer (distributional coverage).

They compared two training schedules: a "Fresh activations" pipeline (one pass through a massive corpus) and a "Buffer replay" pipeline (multiple passes over a smaller stored buffer), as illustrated in the figure below:

Figure 1: Fresh activations and replay at a fixed SAE optimization budget. The replay pipeline generates and stores Ubuf activations once and uses E = Ntotal/Ubuf passes. s denotes bytes per activation. Generation counts refer to token activations. Batching and sequence packing determine the number of calls to the base model.

To formalize this, they adapted a scaling law for limited data (originally proposed by Muennighoff et al. for LLMs). This law calculates an "effective activation count" ($N_{eff}$), which applies a phenomenological discount to repeated data, acknowledging that while replaying data helps optimization, it doesn't provide new information about the feature distribution.

Datasets

  • FineWeb: The primary dataset used for the main scaling grid.
  • FineMath-3plus: Used to test if the findings hold for mathematical text.
  • Stack-Edu Python: Used to test if the findings hold for code.
  • SAEBench: A comprehensive benchmark used for downstream evaluation, including automated interpretability (AutoInterp) and sparse probing.

Key Insights and Ideas

  • The Diversity-Repetition Tradeoff: There is a sharp "elbow" in the performance curve. Initially, increasing the number of unique activations ($U$) significantly improves Explained Variance (EV). However, once the buffer reaches a certain size, the curve plateaus.
  • Dictionary Size Scaling: Larger SAE dictionaries (which have more features) require more unique activations to reach this plateau. The authors found that the saturation interval scales approximately linearly with dictionary size.
  • Universality of Saturation: The plateau effect isn't a fluke of one specific setup. It was validated across:
  • Layers: Early, middle, and late layers of Llama-3.1 8B.
  • Models: Different sizes of the Qwen3 family.
  • Architectures: BatchTopK, standard TopK, and JumpReLU SAEs.
  • Optimization vs. Coverage: The authors argue that unique activations provide coverage (the chance to see a rare feature at all), while total presentations provide optimization (the chance to fit that feature's direction and threshold). Once you have enough coverage, more optimization can be achieved through replay.
Figure 2: Explained variance as a function of unique activation count under fixed total SAE training budgets. With total optimization tokens fixed in each panel, moving right increases activation diversity and decreases the number of epochs over the buffer. EV improves sharply when activation diversity is low and then saturates.

Key Results

  • Massive Storage Savings: By identifying the smallest buffer that reaches the quality plateau, the authors reduced activation storage by 8–64×. For a Llama-3.1 8B model, this meant turning a 9.54 TiB requirement into just 152.6 GiB for a 16k dictionary.
  • Predictive Scaling Law: Their fitted scaling law (RMSE of $4.8 \times 10^{-3}$ EV) successfully predicts the performance of held-out budgets and dictionary widths.
  • Interpretability Preservation: Downstream metrics like AutoInterp and sparse probing accuracy also follow the EV saturation pattern. Replaying a buffer doesn't just result in good reconstruction; it preserves the interpretability of the learned features.
Figure 3: Validation across layers and models (D = 32k, N = 1.28B). Llama-3.1 8B layer sweeps use BatchTopK SAEs, and sweeps across Qwen3 model sizes use TopK SAEs. All curves show the same saturation pattern as diversity increases despite changes in layer, model size, and SAE activation function.

Limitations and Future Work

  • Fixed Sparsity: The main grid held sparsity ($L_0$) fixed. Future work could examine how the diversity requirement changes if one aims for different sparsity levels.
  • Compute vs. Storage: The paper focuses on storage. While replaying a buffer saves disk space, it doesn't necessarily save the compute cost of the SAE forward/backward passes themselves.
  • Modalities: The study was limited to text and code; it remains to be seen if these scaling laws hold for other modalities like vision or audio.
  • Adaptive Buffering: The authors suggest that future research could explore "adaptive buffer refreshing," where the buffer is slowly updated during training rather than being strictly fixed.

Summary updated

Abstract

Activation storage is often a bottleneck in sparse autoencoder (SAE) training. We study how many unique activations an SAE needs when its total training budget is fixed. Quality improves rapidly as the buffer grows, then reaches a plateau where replaying the buffer is nearly as effective as using fresh activations throughout training. Beyond this point, additional unique activations bring little benefit to reconstruction or interpretability, so the buffer need not grow with the training budget. We describe this tradeoff between diversity and repetition with a scaling law for limited data. We validate it across dictionary sizes and token budgets, early, middle, and late layers of Llama-3.1, Qwen3 model sizes, and BatchTopK, TopK, and JumpReLU SAEs. Sweeps using mathematical text and Python code test the effect of changing the activation source. Evaluation covers explained variance (EV), reconstruction MSE, feature absorption, automated interpretability, and sparse probing. These results give a practical training rule: choose the smallest buffer that reaches the quality plateau and replay it for the remaining budget. In our experiments, this reduces activation storage by $8\text{--}64\times$ with negligible quality loss.
BibTeX
@misc{koriagin2026scaling,
  author = {Nikita Koriagin and Nikita Balagansky and Daniil Gavrilov},
  title = {{Scaling Laws for SAE Training Data}},
  year = {2026},
  eprint = {2611.07756},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07756},
}
Figure