Abstract
Massive activations, which manifest in specific feature dimensions of hidden states, introduce a significant bias in large language models (LLMs), leading to an overemphasis on the corresponding token. In this paper, we identify that massive activations originate not from the hidden state but from the intermediate state of a feed-forward network module in an early layer. Expanding on the previous observation that massive activations occur only in specific feature dimensions, we dive deep into the weights that cause massive activations. Specifically, we define top-$k$ massive weights as the weights that contribute to the dimensions with the top-$k$ magnitudes in the intermediate state. When these massive weights are set to zero, the functionality of LLMs is entirely disrupted. However, when all weights except for massive weights are set to zero, it results in a relatively minor performance drop, even though a much larger number of weights are set to zero. This implies that during the pre-training process, learning is dominantly focused on massive weights. Building on this observation, we propose a simple plug-and-play method called MacDrop (massive weights curriculum dropout), to rely less on massive weights during parameter-efficient fine-tuning. This method applies dropout to the pre-trained massive weights, starting with a high dropout probability and gradually decreasing it as fine-tuning progresses. Through various experiments, including zero-shot downstream tasks, long-context tasks, and ablation studies, we demonstrate that MacDrop generally improves performance and strengthens robustness.
Introduction: The "House of Cards" Problem
In the world of Large Language Models (LLMs), "massive activations" are a known phenomenon where certain dimensions in a model's hidden states explode in magnitude. These outliers often act as "attention sinks," causing the model to fixate on specific tokens regardless of the semantic context. While previous research focused on these activations as outputs, this study investigates their origin within the model's weights.
The researchers discovered that these outliers are not random noise but are generated by specific rows in the Feed-Forward Network (FFN) modules of very early layers. Because the model's entire logic eventually leans on these few dimensions, the architecture becomes a "house of cards." If these specific weights are damaged or biased, the whole model collapses. The challenge is to find a way to "de-bias" the model during fine-tuning so it doesn't over-rely on these dominant, pre-trained shortcuts.
Methods: Localizing Massive Weights and MacDrop
The researchers localized "massive weights" by tracing activations back to the intermediate states of FFN modules. In models like Llama-3, these "top-k" massive weights are found in specific rows of the $W_{up}$ and $W_{gate}$ projection matrices in early layers (e.g., Layer 2 or 4).
To address this over-reliance, the authors introduced MacDrop (Massive Weights Curriculum Dropout). Unlike standard dropout applied randomly across all neurons, MacDrop specifically targets identified massive weights during Parameter-Efficient Fine-Tuning (PEFT), such as LoRA or DoRA. The technique follows a curriculum:
- High Initial Dropout: At the start of fine-tuning, massive weights are heavily masked (e.g., 80% dropout), forcing the model to utilize other, less-dominant features.
- Gradual Reduction: The dropout probability linearly decreases to zero as training progresses.
- Stability: By the end of training, the original massive weights are fully restored, ensuring the model retains fundamental capabilities from pre-training.
Key Insights
- The 0.0005% Criticality: In Llama-3-8B, zeroing out just the top-5 massive weights (a mere 0.0005% of total parameters) completely destroys the model's ability to generate coherent text. Conversely, keeping only these 5 rows while zeroing everything else in those matrices allows the model to retain most of its performance.
- The
bosTrigger: The "Beginning of String" (bos) token is the primary trigger for these massive weights. Despite carrying no semantic meaning, it consistently activates these outlier weights, which then propagate through the model via residual connections. - FFN as the Source: While attention sinks manifest in the attention mechanism, they actually originate in the Feed-Forward Networks of early layers.
Experimental Results
The authors evaluated MacDrop across a wide array of benchmarks, including WikiText, ARC, and the LongBench suite. Key findings include:
- Performance Boost: MacDrop consistently improved average zero-shot accuracy. For Llama-3-8B using LoRA, accuracy rose from 76.0% to 76.5%.
- Robustness against Attacks: The authors simulated an "attack" by zeroing out the top-3 massive weights. A standard DoRA-tuned Llama-3-8B collapsed from 76.1% to 40.3% accuracy. However, the model tuned with MacDrop maintained 72.7% accuracy, proving it had learned to be less dependent on those specific weights.
- Long-Context Excellence: MacDrop showed significant gains in long-context scenarios. On the LongBench average, Llama-3-8B improved from 39.03 to 40.54 when MacDrop was added to LoRA.
Limitations and Future Work
MacDrop is highly effective for Llama, Mistral, and Phi-3 families, but provides little benefit for the Gemma-2 family, which does not exhibit the same massive weight dependency due to its unique architecture. While MacDrop excels at objective benchmarks like QA and classification, its impact on open-ended generation tasks was more modest. Future work may investigate whether architectural changes during initial pre-training could prevent these "house of cards" weights from forming entirely.
Summary updated