TL;DR
As language models grow, they naturally rely less on single-purpose "monosemantic" neurons and more on multi-functional "polysemantic" representations. This paper introduces Learning Towards Emergence (L2E), a scalable method to actively penalize and inhibit monosemantic neurons during LLM pretraining. By combining a moving threshold for fast neuron selection and a regularization-based loss, L2E improves downstream reasoning performance on Pythia models up to 2.8B parameters with negligible computational overhead.
What Problem Does This Paper Address?
When large language models cross certain parameter thresholds, they often exhibit sharp leaps in reasoning ability—a phenomenon known as "emergence." Mechanistic interpretability research has shown an intriguing correlation: larger models contain far fewer monosemantic neurons (neurons that activate for only one specific feature, like the word "Python" or "Dutch text") and far more polysemantic neurons (neurons responding to diverse contexts).
Prior work tried actively suppressing monosemantic neurons during fine-tuning on tiny models ($\le 88\text{M}$ parameters) to simulate emergence. However, extending this to pretraining large models failed due to three major hurdles:
- Identifying which and how many neurons to suppress lacked a principled, scale-invariant rule.
- Exact top-$k$ neuron sorting across millions of parameters caused steep computational bottlenecks ($\mathcal{O}(N \log N)$).
- Existing deactivation methods assumed neurons were already well-trained, which backfires during early pretraining when weights are noisy.
Methods
L2E addresses these hurdles with three key innovations:
- False Killing Rate (FKR): The authors define FKR to measure the risk of inadvertently suppressing a generalist/polysemantic neuron on unrelated inputs. Analyzing FKR across model sizes revealed a consistent, scale-invariant sweet spot: suppressing the top 2% most monosemantic activations minimized collateral damage.
- Moving Threshold for Fast Retrieval: Instead of running expensive exact top-$k$ sorting in every batch, L2E maintains a running scalar threshold $\tau^*$. It performs an $\mathcal{O}(1)$ negative-feedback update per step: if too many neurons exceed $\tau^*$, the threshold rises, and vice-versa. This reduces neuron retrieval to simple element-wise comparisons.
- Regularization-Style Inhibition: Rather than hard-zeroing activations, L2E introduces an auxiliary loss penalty directly targeting excessive activation variance: $$\mathcal{L}_{\text{MS}} = \log \left(\theta_\omega(x^{[i]}) - \bar{z}\right)^2$$ This acts as a gentle regularizer suitable from step 0 of pretraining, stabilizing training dynamics without disrupting unformed circuits.
In practice, L2E is applied to the outputs of transformer blocks in the middle two layers of the model, where abstract reasoning representations predominantly reside.
Datasets
- Feature Datasets (for probing & validation): Natural Language (Europarl multi-lingual data), Data Subset (The Pile domains), and Code Language (programming languages).
- Pretraining Data: The deduplicated Pile dataset (trained for 14.3k steps, ~10% of standard Pythia pretraining schedules due to compute budgets).
- Evaluation Benchmarks: ARC-Challenge, PIQA, and SciQ (evaluated under 0-shot and 5-shot settings using the LM Evaluation Harness), alongside ARC-Easy for task-complexity analysis.
Key Insights and Ideas
- Monosemanticity decreases with scale and layer depth: Probing Pythia models from 70M to 6.9B parameters via Kolmogorov-Smirnov (K-S) tests confirmed that monosemanticity steadily declines as models scale up. Across layers, middle layers exhibit the lowest monosemanticity, suggesting higher-level reasoning relies heavily on distributed, polysemantic circuits.
- The "Rote Memorization" Trade-Off: In an ablation comparing ARC-Challenge vs. ARC-Easy, L2E consistently boosted performance on ARC-Challenge but degraded performance on ARC-Easy. This suggests monosemanticity behaves like rote memorization: helpful for shallow pattern recall, but actively detrimental to complex, multi-step generalization.
- Coexistence with Sparsity: While reducing monosemanticity encourages distributed representations, the authors note it can still coexist with activation/weight pruning and Mixture-of-Experts (MoE) architectures by targeting only the top extreme outliers.
Key Results
- Reasoning Performance Gains: Evaluated on Pythia (70M, 410M, and 2.8B), L2E consistently outperformed both vanilla Pythia and a standard Dropout baseline across ARC-C, PIQA, and SciQ:
- 0-shot: Achieved an average +2.0% gain over vanilla Pythia and +1.3% over Dropout.
- 5-shot: Achieved an average +2.3% gain over vanilla Pythia.
- Computational Efficiency:
- At the 70M scale, previous methods (MEmeL) added +9.7% runtime overhead, while L2E added +3.4%.
- At the 2.8B scale, MEmeL’s overhead ballooned to +27.0%, whereas L2E dropped to just +1.48% because element-wise thresholding was easily amortized across attention calculations.
- Middle Layers Matter Most: Restricting suppression to the middle layers outperformed suppressing bottom or top layers (e.g., on 410M 0-shot, middle layers scored ~1.7–2.2% improvements compared to 1.1% on Top/Bottom).
Limitations and Future Work
- Scale and Training Length: Experiments were capped at Pythia-2.8B trained for only 10% of total pretraining tokens due to GPU constraints (24 $\times$ H100s). Testing full pretraining runs up to 7B+ parameters remains necessary to definitively confirm emergent phase shifts.
- Dependence on Feature Datasets: Current definitions of monosemanticity rely heavily on hand-crafted feature probes. The authors noted high variance across feature sets (e.g., Code Language was heavily biased towards Python and yielded noisy probe classifiers). Establishing standardized, rigorous feature sets is an essential open direction.
Summary updated