Multi-Layer Attention is the Amplifier of Demonstration Effectiveness

Dingzirui Wang, Xuangliang Zhang, Keyan Xu, Qingfu Zhu, Wanxiang Che, Yang DengarXiv 2025 · DOI · arXiv

Introduction

In-Context Learning (ICL) has revolutionized how we interact with Large Language Models (LLMs), allowing them to perform new tasks using only a few examples provided in the prompt. However, the quality of these demonstrations is highly variable. Some examples are redundant—containing information the model already internalized during pre-training—while others are simply noisy or irrelevant to the user's specific query. Traditional selection methods, such as BM25 or cosine similarity, often fail because they focus solely on surface-level text similarity rather than the model's internal state.

Conceptual overview of how multi-layer attention amplifies effective demonstrations while filtering ineffective ones.

This research introduces a novel perspective: multi-layer attention acts as an amplifier. As information passes through the Transformer's layers, the model progressively prioritizes useful demonstrations and filters out noise, ensuring that the final output is driven by the most relevant context.

Theoretical Framework: The Amplifier Theorem

The authors formalize this intuition using a theoretical framework based on Linear Self-Attention (LSA) and Gradient Flow. By defining the effectiveness of a demonstration as the magnitude of its gradient flow toward the final output, they provide a mathematical basis for why certain examples drive better performance.

The core contribution is the Amplifier Theorem, which proves that in a multi-layer Transformer, the disparity between effective and ineffective demonstrations is magnified as data moves from the input layer toward the output. Essentially, the deeper layers of the model act as a lens, focusing on the most relevant and informative signals while suppressing the "redundancy trap"—information the model already knows.

GRADS: Gradient-flow-based Selection

Building on this theory, the researchers propose GRADS (Gradient-flow-based Demonstration Selection). Unlike similarity-based methods, GRADS selects examples by calculating the magnitude of their gradient flow relative to a specific query. To ensure practical utility, the method uses pre-computed demonstration encodings, making the selection process nearly as fast as standard 1-shot inference.

Empirical Results and Discussion

The effectiveness of GRADS was validated across five benchmarks (GSM8K, MATH, ARC-Challenge, MMLU-Pro, and Amazon Review) using four state-of-the-art models: Llama2-7b, Llama3.1-8b, Llama-R1-8b, and Qwen3-8b.

Key Findings

  1. Performance Gains: GRADS achieved a 6.8% relative improvement over strong baselines like MMR and MoD. The boost was particularly pronounced in complex reasoning tasks (MATH and MMLU-Pro), where selecting the right knowledge is critical.
  2. Validation of the Amplifier Effect: By tracking gradient flow across layers, the authors empirically confirmed their theorem. As shown in the figure below, the ratio of useful to useless signal (represented by the green line) consistently increases as information moves deeper into the model.
Empirical evidence across multiple LLMs showing the amplification of the signal-to-noise ratio (green line) as a function of layer depth.
  1. Relevance vs. Knowledge: The research highlights that effectiveness is a dual-factor equation: how relevant a demo is to the query, and how much new information it provides. GRADS is the first method to explicitly account for both by leveraging the model's internal gradients.

Conclusion and Future Work

This study shifts the focus of demonstration selection from external similarity to internal model dynamics. While the current proofs rely on Linear Self-Attention, the empirical results across various modern architectures suggest the "amplifier" effect is a robust property of multi-layer Transformers. Future research will explore how this effect scales with multi-shot prompts and how non-linearities like Softmax attention influence the amplification process.

Summary updated

Abstract

Numerous studies have investigated the underlying mechanisms of in-context learning (ICL) effectiveness to inspire the design of related methods. However, existing work predominantly assumes the effectiveness of the demonstrations provided within ICL, while many research indicates that not all demonstrations are effective, failing to yielding any performance improvement during ICL. Therefore, in this paper, we investigate the reasons behind demonstration ineffectiveness. Our analysis is based on gradient flow and linear self-attention models. By setting the gradient flow to zero, we deduce that a demonstration becomes ineffective if its information has either been learned by the model or is irrelevant to the user query. Furthermore, we demonstrate that in multi-layer models, the disparity in effectiveness among demonstrations is amplified with layer increasing, causing the model to focus more on effective ones. Considering that current demonstration selection methods primarily focus on the relevance to the user query while overlooking the information that the model has already assimilated, we propose a novel method called GradS, which leverages gradient flow for demonstration selection. We use the magnitude of the gradient flow of the demonstration with respect to a given user query as the criterion, thereby ensuring the effectiveness of the chosen ones. We validate our derivation and GradS on four prominent LLMs across five mainstream datasets. The experimental results confirm that the disparity in effectiveness among demonstrations is magnified as the model layer increases, substantiating our derivations. Moreover, GradS achieves a relative improvement of $6.8\%$ on average over the strongest baselines, demonstrating its effectiveness.
BibTeX
@misc{wang2025multi,
  author = {Dingzirui Wang and Xuangliang Zhang and Keyan Xu and Qingfu Zhu and Wanxiang Che and Yang Deng},
  title = {{Multi-Layer Attention is the Amplifier of Demonstration Effectiveness}},
  year = {2025},
  eprint = {2508.00385},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.00385},
}
Figure