House of Cards: Massive Weights in LLMs

Jaehoon Oh, Seungjun Shin, Dokwan OhSubmission to ICML 2025 · Rejected

TL;DR

Large language models (LLMs) appear formidable, but their capabilities hinge precariously on an astonishingly tiny subset of parameters—dubbed massive weights—located in an early feed-forward layer. Zeroing out as few as five rows of these weights (~0.0005% of the model) completely destroys model performance, while keeping only these weights preserves most capabilities. To mitigate this over-reliance during parameter-efficient fine-tuning (PEFT), the authors introduce MacDrop, a curriculum weight-dropout technique that improves downstream zero-shot accuracy, long-context handling, and adversarial robustness.

What Problem Does This Paper Address?

Recent work revealed two closely linked phenomena in LLMs: attention sinks (where models disproportionately attend to the initial token or delimiter tokens regardless of semantic value) and massive activations (anomalously large hidden-state activation magnitudes concentrated in a few fixed feature dimensions).

While prior work treated these as hidden-state artifacts, this paper investigates their mechanistic origin in the weight space:

  • Exactly where in the network do massive activations originate?
  • How much do modern LLMs actually depend on the specific weights generating them?
  • Can we prevent downstream fine-tuning from becoming overly dependent on these dominant pre-trained pathways?

Methods

  1. Pinpointing Origin via the bos Token: By feeding the beginning-of-sequence (bos) token, the authors observe that massive activations reliably trigger across architectures. Tracing intermediate representations reveals that massive activations do not originate in hidden states or self-attention modules, but in the intermediate state of a feed-forward network (FFN) at an early layer: $$h^{\text{inter}}_l = \sigma(W_{\text{gate}} x) \odot W_{\text{up}} x$$ Once triggered, they propagate through subsequent layers almost purely via residual skip connections.

  2. Top-$k$ Massive Weights Definition & Attack: The authors define the top-$k$ massive weights as the specific rows of $W_{\text{up}}$ (and $W_{\text{gate}}$) at layer $l$ corresponding to the top-$k$ magnitude values of $h^{\text{inter}}_l$. They test model vulnerability using two opposite interventions:

  3. Top-$k$ Zeroing: Setting these top-$k$ rows to zero.
  4. Top-$k$ Retaining: Setting all other rows in those matrices to zero while preserving only the top-$k$ massive weights.
  5. MacDrop (Massive Weights Curriculum Dropout): During PEFT (such as LoRA or DoRA), instead of dropping adapter weights, MacDrop applies dropout directly to the frozen pre-trained massive weights. It begins with a high dropout probability $p_0$ and linearly decays it to 0 as training steps progress: $$p = p_0 \times \left(1 - \frac{t_{\text{step}}}{T_{\text{step}}}\right)$$ This forces the adapter to learn representations independent of the dominating massive weights early in training, while gradually restoring the undamaged pre-trained model for final inference.
       ┌────────────────────────┐
       │ Pre-trained FFN Weights│
       │    (W_gate / W_up)     │
       └───────────┬────────────┘
                   │
         [ Identify Top-k Rows ]  <-- Using <s> (bos) token
                   │
                   ▼
       ┌────────────────────────┐
       │   Top-k Massive Rows   │
       └───────────┬────────────┘
                   │
   Early Steps:    │ High Dropout (p ≈ 0.8)  --> Forces adapter to learn
   Late Steps:     │ Decays to p = 0         --> Restores base capability
                   ▼
       ┌────────────────────────┐
       │ Trainable Adapter (LoRA)│
       └────────────────────────┘

Datasets

  • Instruction Fine-Tuning: alpaca_gpt4_en (3 epochs).
  • Perplexity Evaluation: WikiText, C4, and PG-19.
  • Zero-Shot Downstream Tasks: ARC-Easy, ARC-Challenge, BoolQ, PIQA, and WinoGrande.
  • Long-Context Benchmark: LongBench (16 English datasets across single/multi-document QA, summarization, few-shot, synthetic, and code tasks).
  • Text Generation Benchmark: Spec-Bench (multi-turn conversation, translation, summarization, QA, GSM8K math reasoning, and RAG), evaluated via a 5-point Likert scale using Prometheus-2-7B.

Key Insights and Ideas

  • The "House of Cards" Effect: In Llama-3-8B, zeroing just the top-5 massive weight rows (0.0005% of parameters in layer 2) causes average perplexity to explode from 8.22 to 122.74, and zero-shot accuracy plunges to near chance (73.0% $\to$ 39.4%). Conversely, keeping only those 5 rows while zeroing thousands of other rows in that layer yields an average accuracy of 71.0%. LLMs depend far more on these isolated parameters than on the rest of that layer combined.
  • Architectural Immunity:
  • Gemma-2 does not exhibit massive activations or initial attention sinks. Its architecture includes extra post-attention and post-FFN LayerNorm modules, rendering it completely immune to top-$k$ zeroing attacks.
  • Phi-3-medium shows high resistance, likely because it incorporated dropout across module outputs combined with extensive token pre-training.
  • MoE Routing Bias: In Mixtral-8x7B, massive activations at layer 2 cause router probabilities to heavily collapse onto a single expert (Expert 4). Zeroing the massive weights in this single expert severely damages the model.

Key Results

  • Zero-Shot Improvements: Fine-tuning with MacDrop consistently improves downstream accuracy on Llama-3-8B (LoRA: 76.0% $\to$ 76.5%; DoRA: 76.1% $\to$ 76.4%) and Mistral-7B (LoRA: 75.3% $\to$ 76.2%), showing notable gains on reasoning benchmarks like ARC-Easy (+3.3% on Llama-3-8B).
  • Substantial Attack Robustness: Under a top-3 zeroing attack, standard fine-tuned models collapse (~40.3% accuracy). When trained with DoRA + MacDrop, Llama-3-8B retains 72.7% accuracy, demonstrating that the learned adapter successfully circumvents dependence on massive weights.
  • Long Contexts: MacDrop improves average LongBench scores across both Llama-3-8B (39.03 $\to$ 40.54) and Mistral-7B (36.91 $\to$ 38.03).

Limitations and Future Work

  • Limited Generation Gains: MacDrop produced negligible improvements on open-ended generation tasks evaluated by Prometheus-2-7B on Spec-Bench (e.g., Llama-3-8B LoRA averaged 3.82 vs. 3.81 with MacDrop).
  • Architecture Dependency: The technique is ineffective on models that do not natively rely on massive weights, such as Gemma-2 and Phi-3-medium.
  • Future Directions: The authors highlight extending weight-space massive weight analysis to model merging, model editing, and activation-aware pruning/quantization routines.

Summary updated

Abstract

Massive activations, which manifest in specific feature dimensions of hidden states, introduce a significant bias in large language models (LLMs), leading to an overemphasis on the corresponding token. In this paper, we identify that massive activations originate not from the hidden state but from the intermediate state of a feed-forward network module in an early layer. Expanding on the previous observation that massive activations occur only in specific feature dimensions, we dive deep into the weights that cause massive activations. Specifically, we define top-$k$ massive weights as the weights that contribute to the dimensions with the top-$k$ magnitudes in the intermediate state. When these massive weights are set to zero, the functionality of LLMs is entirely disrupted. However, when all weights except for massive weights are set to zero, it results in a relatively minor performance drop, even though a much larger number of weights are set to zero. This implies that during the pre-training process, learning is dominantly focused on massive weights. Building on this observation, we propose a simple plug-and-play method called MacDrop (massive weights curriculum dropout), to rely less on massive weights during parameter-efficient fine-tuning. This method applies dropout to the pre-trained massive weights, starting with a high dropout probability and gradually decreasing it as fine-tuning progresses. Through various experiments, including zero-shot downstream tasks, long-context tasks, and ablation studies, we demonstrate that MacDrop generally improves performance and robustness.
BibTeX
@misc{oh2024house,
  author = {Jaehoon Oh and Seungjun Shin and Dokwan Oh},
  title = {{House of Cards: Massive Weights in LLMs}},
  year = {2024},
  eprint = {2410.01866},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.01866},
}

Loading related content...

Figure