TL;DR
Large language models (LLMs) appear formidable, but their capabilities hinge precariously on an astonishingly tiny subset of parameters—dubbed massive weights—located in an early feed-forward layer. Zeroing out as few as five rows of these weights (~0.0005% of the model) completely destroys model performance, while keeping only these weights preserves most capabilities. To mitigate this over-reliance during parameter-efficient fine-tuning (PEFT), the authors introduce MacDrop, a curriculum weight-dropout technique that improves downstream zero-shot accuracy, long-context handling, and adversarial robustness.
What Problem Does This Paper Address?
Recent work revealed two closely linked phenomena in LLMs: attention sinks (where models disproportionately attend to the initial token or delimiter tokens regardless of semantic value) and massive activations (anomalously large hidden-state activation magnitudes concentrated in a few fixed feature dimensions).
While prior work treated these as hidden-state artifacts, this paper investigates their mechanistic origin in the weight space:
- Exactly where in the network do massive activations originate?
- How much do modern LLMs actually depend on the specific weights generating them?
- Can we prevent downstream fine-tuning from becoming overly dependent on these dominant pre-trained pathways?
Methods
-
Pinpointing Origin via the
bosToken: By feeding the beginning-of-sequence (bos) token, the authors observe that massive activations reliably trigger across architectures. Tracing intermediate representations reveals that massive activations do not originate in hidden states or self-attention modules, but in the intermediate state of a feed-forward network (FFN) at an early layer: $$h^{\text{inter}}_l = \sigma(W_{\text{gate}} x) \odot W_{\text{up}} x$$ Once triggered, they propagate through subsequent layers almost purely via residual skip connections. -
Top-$k$ Massive Weights Definition & Attack: The authors define the top-$k$ massive weights as the specific rows of $W_{\text{up}}$ (and $W_{\text{gate}}$) at layer $l$ corresponding to the top-$k$ magnitude values of $h^{\text{inter}}_l$. They test model vulnerability using two opposite interventions:
- Top-$k$ Zeroing: Setting these top-$k$ rows to zero.
- Top-$k$ Retaining: Setting all other rows in those matrices to zero while preserving only the top-$k$ massive weights.
- MacDrop (Massive Weights Curriculum Dropout): During PEFT (such as LoRA or DoRA), instead of dropping adapter weights, MacDrop applies dropout directly to the frozen pre-trained massive weights. It begins with a high dropout probability $p_0$ and linearly decays it to 0 as training steps progress: $$p = p_0 \times \left(1 - \frac{t_{\text{step}}}{T_{\text{step}}}\right)$$ This forces the adapter to learn representations independent of the dominating massive weights early in training, while gradually restoring the undamaged pre-trained model for final inference.
┌────────────────────────┐
│ Pre-trained FFN Weights│
│ (W_gate / W_up) │
└───────────┬────────────┘
│
[ Identify Top-k Rows ] <-- Using <s> (bos) token
│
▼
┌────────────────────────┐
│ Top-k Massive Rows │
└───────────┬────────────┘
│
Early Steps: │ High Dropout (p ≈ 0.8) --> Forces adapter to learn
Late Steps: │ Decays to p = 0 --> Restores base capability
▼
┌────────────────────────┐
│ Trainable Adapter (LoRA)│
└────────────────────────┘
Datasets
- Instruction Fine-Tuning:
alpaca_gpt4_en(3 epochs). - Perplexity Evaluation: WikiText, C4, and PG-19.
- Zero-Shot Downstream Tasks: ARC-Easy, ARC-Challenge, BoolQ, PIQA, and WinoGrande.
- Long-Context Benchmark: LongBench (16 English datasets across single/multi-document QA, summarization, few-shot, synthetic, and code tasks).
- Text Generation Benchmark: Spec-Bench (multi-turn conversation, translation, summarization, QA, GSM8K math reasoning, and RAG), evaluated via a 5-point Likert scale using Prometheus-2-7B.
Key Insights and Ideas
- The "House of Cards" Effect: In Llama-3-8B, zeroing just the top-5 massive weight rows (0.0005% of parameters in layer 2) causes average perplexity to explode from 8.22 to 122.74, and zero-shot accuracy plunges to near chance (73.0% $\to$ 39.4%). Conversely, keeping only those 5 rows while zeroing thousands of other rows in that layer yields an average accuracy of 71.0%. LLMs depend far more on these isolated parameters than on the rest of that layer combined.
- Architectural Immunity:
- Gemma-2 does not exhibit massive activations or initial attention sinks. Its architecture includes extra post-attention and post-FFN LayerNorm modules, rendering it completely immune to top-$k$ zeroing attacks.
- Phi-3-medium shows high resistance, likely because it incorporated dropout across module outputs combined with extensive token pre-training.
- MoE Routing Bias: In Mixtral-8x7B, massive activations at layer 2 cause router probabilities to heavily collapse onto a single expert (Expert 4). Zeroing the massive weights in this single expert severely damages the model.
Key Results
- Zero-Shot Improvements: Fine-tuning with MacDrop consistently improves downstream accuracy on Llama-3-8B (LoRA: 76.0% $\to$ 76.5%; DoRA: 76.1% $\to$ 76.4%) and Mistral-7B (LoRA: 75.3% $\to$ 76.2%), showing notable gains on reasoning benchmarks like ARC-Easy (+3.3% on Llama-3-8B).
- Substantial Attack Robustness: Under a top-3 zeroing attack, standard fine-tuned models collapse (~40.3% accuracy). When trained with DoRA + MacDrop, Llama-3-8B retains 72.7% accuracy, demonstrating that the learned adapter successfully circumvents dependence on massive weights.
- Long Contexts: MacDrop improves average LongBench scores across both Llama-3-8B (39.03 $\to$ 40.54) and Mistral-7B (36.91 $\to$ 38.03).
Limitations and Future Work
- Limited Generation Gains: MacDrop produced negligible improvements on open-ended generation tasks evaluated by Prometheus-2-7B on Spec-Bench (e.g., Llama-3-8B LoRA averaged 3.82 vs. 3.81 with MacDrop).
- Architecture Dependency: The technique is ineffective on models that do not natively rely on massive weights, such as Gemma-2 and Phi-3-medium.
- Future Directions: The authors highlight extending weight-space massive weight analysis to model merging, model editing, and activation-aware pruning/quantization routines.
Summary updated