TL;DR
Adjacent layers in transformer models often seem to "fight" each other, writing opposing signals into the residual stream. This paper reveals that in gated models (like Llama or Mistral), this isn't a sophisticated "propose-and-reject" mechanism for features, but rather a mathematical side effect of "leaky" closed gates in SiLU and GELU activation functions that cause a general contraction of the input space.
What Problem Does This Paper Address?
In modern Large Language Models (LLMs), the residual stream acts as a communication channel where each layer adds its own "write." Curiously, researchers recently observed that adjacent layers often write against each other—if layer $i$ writes in one direction, layer $i+1$ often writes in the opposite direction (negative cosine similarity).
The leading hypothesis was a "propose-and-reject" framework: one layer proposes candidate features, and the next layer selectively scrubs the inappropriate ones. This paper investigates whether this cancellation is truly a selective "rejection" of specific features or just a byproduct of the model's architecture.
Methods
The authors introduce a counterfactual decomposition to analyze what a layer writes along the direction of the previous layer's contribution ($v$). They split the next layer's output into three components:
- Attention ($A$): What the attention mechanism writes along $v$.
- Co-writing ($C$): What the MLP would have written along $v$ even if $v$ weren't there.
- Damping ($D$): How much less the MLP writes along $v$ specifically because $v$ is present in its input.
Cancellation occurs whenever the Damping ($D$) outweighs the sum of Attention ($A$) and Co-writing ($C$).
The authors also performed interventions, such as annealing the negative region of gate activations (SiLU/GELU) during continued pretraining to see if the cancellation effect persisted when the "leaky" part of the gate was removed.
Datasets
The study evaluated 16 different models, including:
- Gated Models: Llama-3.2-1B, Qwen2.5-0.5B, Mistral-7B, Gemma-2B, and OLMo-1B.
- Non-gated Models: Pythia-160m, GPT-2, and OPT-1.3B.
- Datasets: Measurements were primarily taken using the FineWeb-Edu corpus and the WikiText-103 test set.
Key Insights and Ideas
- Closed Gates Leak: In gated MLPs using SiLU or GELU, a "closed" gate isn't actually zero; it's slightly negative. Because of this, neurons whose output weights align with their input weights end up subtracting a fraction of whatever they read. This creates a general "contraction" or damping of the input.
- It’s Not Selective: The authors found that the MLP damps the previous layer's write less than it damps writes from other tokens or random directions. This strongly suggests the model isn't "singling out" specific features to reject; it's just damping everything it sees.
- Architecture vs. Learning: The cancellation is driven by the negative "lobe" of the activation function. When the authors trained models from scratch or annealed existing models to remove this negative region, the cancellation effect vanished.
- Training Dynamics: Damping grows significantly during pretraining while co-writing stays relatively flat. Cancellation only begins once the damping effect becomes strong enough to overtake the other components.
Key Results
- Damping Dominates: In models that cancel (like Llama-3.2-1B and Qwen2.5-0.5B), damping exceeds attention and co-writing by 0.135 to 0.244 units per unit of the previous write.
- Intervention Success: Annealing the negative region of Qwen2.5-0.5B's gate activation moved the window cosine from -0.184 to +0.021, effectively stopping the cancellation.
- Non-gated Contrast: Non-gated models like Pythia also cancel, but they do it differently—through "active" neurons whose output weights directly oppose their input weights, rather than through leaky gates.
Limitations and Future Work
- Scale: The study focused on models up to 8B parameters. Whether these dynamics hold or change in frontier-scale models (100B+) remains to be seen.
- Feature Selectivity: While the authors showed that directions aren't singled out, they acknowledge that their tests don't rule out selectivity at the level of a "feature dictionary" (e.g., via Sparse Autoencoders).
- Optimization: It is currently unknown if this architectural damping provides a hidden benefit to optimization during training, despite the slight loss in performance observed when it was removed during annealing.
- Co-writing/Damping Correlation: The paper notes that co-writing and damping rise together during training but leaves the explanation for this correlation as an open question.
Summary updated