Learning Towards Emergence: Paving the Way to Induce Emergence by Inhibiting Monosemantic Neurons on Pre-trained Models

Jiachuan Wang, Shimin Di, Tianhao Tang, Haoyang Li, Charles Wang-wai Ng, Xiaofang Zhou, Lei ChenSubmission to ICLR 2025 · Rejected

TL;DR

As language models grow, they naturally rely less on single-purpose "monosemantic" neurons and more on multi-functional "polysemantic" representations. This paper introduces Learning Towards Emergence (L2E), a scalable method to actively penalize and inhibit monosemantic neurons during LLM pretraining. By combining a moving threshold for fast neuron selection and a regularization-based loss, L2E improves downstream reasoning performance on Pythia models up to 2.8B parameters with negligible computational overhead.

What Problem Does This Paper Address?

When large language models cross certain parameter thresholds, they often exhibit sharp leaps in reasoning ability—a phenomenon known as "emergence." Mechanistic interpretability research has shown an intriguing correlation: larger models contain far fewer monosemantic neurons (neurons that activate for only one specific feature, like the word "Python" or "Dutch text") and far more polysemantic neurons (neurons responding to diverse contexts).

Prior work tried actively suppressing monosemantic neurons during fine-tuning on tiny models ($\le 88\text{M}$ parameters) to simulate emergence. However, extending this to pretraining large models failed due to three major hurdles:

  1. Identifying which and how many neurons to suppress lacked a principled, scale-invariant rule.
  2. Exact top-$k$ neuron sorting across millions of parameters caused steep computational bottlenecks ($\mathcal{O}(N \log N)$).
  3. Existing deactivation methods assumed neurons were already well-trained, which backfires during early pretraining when weights are noisy.

Methods

L2E addresses these hurdles with three key innovations:

  • False Killing Rate (FKR): The authors define FKR to measure the risk of inadvertently suppressing a generalist/polysemantic neuron on unrelated inputs. Analyzing FKR across model sizes revealed a consistent, scale-invariant sweet spot: suppressing the top 2% most monosemantic activations minimized collateral damage.
  • Moving Threshold for Fast Retrieval: Instead of running expensive exact top-$k$ sorting in every batch, L2E maintains a running scalar threshold $\tau^*$. It performs an $\mathcal{O}(1)$ negative-feedback update per step: if too many neurons exceed $\tau^*$, the threshold rises, and vice-versa. This reduces neuron retrieval to simple element-wise comparisons.
  • Regularization-Style Inhibition: Rather than hard-zeroing activations, L2E introduces an auxiliary loss penalty directly targeting excessive activation variance: $$\mathcal{L}_{\text{MS}} = \log \left(\theta_\omega(x^{[i]}) - \bar{z}\right)^2$$ This acts as a gentle regularizer suitable from step 0 of pretraining, stabilizing training dynamics without disrupting unformed circuits.

In practice, L2E is applied to the outputs of transformer blocks in the middle two layers of the model, where abstract reasoning representations predominantly reside.

Datasets

  • Feature Datasets (for probing & validation): Natural Language (Europarl multi-lingual data), Data Subset (The Pile domains), and Code Language (programming languages).
  • Pretraining Data: The deduplicated Pile dataset (trained for 14.3k steps, ~10% of standard Pythia pretraining schedules due to compute budgets).
  • Evaluation Benchmarks: ARC-Challenge, PIQA, and SciQ (evaluated under 0-shot and 5-shot settings using the LM Evaluation Harness), alongside ARC-Easy for task-complexity analysis.

Key Insights and Ideas

  • Monosemanticity decreases with scale and layer depth: Probing Pythia models from 70M to 6.9B parameters via Kolmogorov-Smirnov (K-S) tests confirmed that monosemanticity steadily declines as models scale up. Across layers, middle layers exhibit the lowest monosemanticity, suggesting higher-level reasoning relies heavily on distributed, polysemantic circuits.
  • The "Rote Memorization" Trade-Off: In an ablation comparing ARC-Challenge vs. ARC-Easy, L2E consistently boosted performance on ARC-Challenge but degraded performance on ARC-Easy. This suggests monosemanticity behaves like rote memorization: helpful for shallow pattern recall, but actively detrimental to complex, multi-step generalization.
  • Coexistence with Sparsity: While reducing monosemanticity encourages distributed representations, the authors note it can still coexist with activation/weight pruning and Mixture-of-Experts (MoE) architectures by targeting only the top extreme outliers.

Key Results

  • Reasoning Performance Gains: Evaluated on Pythia (70M, 410M, and 2.8B), L2E consistently outperformed both vanilla Pythia and a standard Dropout baseline across ARC-C, PIQA, and SciQ:
  • 0-shot: Achieved an average +2.0% gain over vanilla Pythia and +1.3% over Dropout.
  • 5-shot: Achieved an average +2.3% gain over vanilla Pythia.
  • Computational Efficiency:
  • At the 70M scale, previous methods (MEmeL) added +9.7% runtime overhead, while L2E added +3.4%.
  • At the 2.8B scale, MEmeL’s overhead ballooned to +27.0%, whereas L2E dropped to just +1.48% because element-wise thresholding was easily amortized across attention calculations.
  • Middle Layers Matter Most: Restricting suppression to the middle layers outperformed suppressing bottom or top layers (e.g., on 410M 0-shot, middle layers scored ~1.7–2.2% improvements compared to 1.1% on Top/Bottom).

Limitations and Future Work

  • Scale and Training Length: Experiments were capped at Pythia-2.8B trained for only 10% of total pretraining tokens due to GPU constraints (24 $\times$ H100s). Testing full pretraining runs up to 7B+ parameters remains necessary to definitively confirm emergent phase shifts.
  • Dependence on Feature Datasets: Current definitions of monosemanticity rely heavily on hand-crafted feature probes. The authors noted high variance across feature sets (e.g., Code Language was heavily biased towards Python and yielded noisy probe classifiers). Establishing standardized, rigorous feature sets is an essential open direction.

Summary updated

Abstract

Emergence, the phenomenon of a rapid performance increase once the model scale reaches a threshold, has achieved widespread attention recently. The literature has observed that monosemantic neurons in neural networks gradually diminish as the model scale increases. Subsequently, *Learning From Emergence* is proposed to actively inhibit monosemantic neurons in relatively small neural networks (e.g., BERT and Swin-Transformer) for promoting model performance with fine-tuning. However, to ultimately achieve emergence, it is demanding to support the monosemantic neuron inhibition in the pretraining phase of large-scale models. Thus, this work further pushes the boundary of this research direction to be *Learning Towards Emergence (L2E)* and enables the training and validating of the impact of inhibiting monosemantic neurons on larger pre-trained neural networks (e.g., Pythia-70M, 410M, and 2.8B). More specifically, to bridge the gap in current research, we first conduct experiments on models of various scales (up to 6.9B) to validate the monosemantic ideas. Then, we present a novel method L2E to address the inefficient monosemantic neuron retrieval and ineffective monosemantic neuron inhibition when existing methods are applied in the pretraining phase of large-scale models. It employs an adjustable thresholding technique for efficient neuron retrieval, incorporates a False Killing Rate metric to assess inhibition effects, and proposes a regularization-style inhibition approach, which addresses the limitations of previous approaches in both efficiency and effectiveness.
BibTeX
@misc{wang2025learning,
  author = {Jiachuan Wang and Shimin Di and Tianhao Tang and Haoyang LI and Charles Wang-wai Ng and Xiaofang Zhou and Lei Chen},
  title = {{Learning Towards Emergence: Paving the Way to Induce Emergence by Inhibiting Monosemantic Neurons on Pre-trained Models}},
  year = {2025},
  eprint = {2503.23298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.23298},
}
Figure