In standard Transformer architectures, attention is calculated from scratch at every single layer. The query, key, and value matrices are projected from the current layer's hidden states, completely oblivious to the attention patterns computed just a step prior. This "memoryless" approach forces deep networks to repeatedly rediscover basic structural or spatial alignments, introducing noise and wasting parameter capacity.
Iterative attention refinement changes this paradigm. Instead of treating each layer's attention as an isolated calculation, it treats attention as an evolving, multi-stage optimization process. By carrying attention maps forward through the network, models can treat previous attention distributions as prior beliefs, progressively sharpening their focus from a broad, coarse overview to a highly precise, fine-grained alignment.
Inter-Layer Attention Propagation and Residuals
The most direct way to implement this idea is to establish a continuous path for attention scores across layers, mirroring how residual connections function for feature embeddings.
In RealFormer: Transformer Likes Residual Attention, Ruining He et al. add the pre-softmax attention scores from the previous layer directly to the current layer's scores. This simple addition creates a direct highway for contextual relationships to propagate. Similarly, EfficientASR: Speech Recognition Network Compression via Attention Redundancy and Chunk-Level FFN Optimization leverages residual updates to recycle attention scores across layers, allowing the network to refine its focus while significantly reducing parameter counts.
Rather than simple addition, some architectures use multiplicative gating to inherit historical focus. Wei Long demonstrates in Progressive Focused Transformer for Single Image Super-Resolution that performing an element-wise multiplication between the current layer's candidate attention map and the preceding layer's map reinforces consistently relevant features while suppressing transient noise. This cumulative inheritance ensures that deep layers do not lose track of global structural constraints, a concept also utilized by Zhaoyu Zhang et al. in ERNIE-RNA: an RNA language model with structure-enhanced representations to progressively refine structural base-pairing priors.
Coarse-to-Fine Recurrent Refinement
Beyond stacking distinct layers, iterative refinement can be achieved by feeding features back through a recurrent loop or cascading sequential decoding stages. This mimics the human visual system's ability to "squint" or look longer at a complex scene to discern subtle details.
In Focus Longer to See Better: Recursively Refined Attention for Fine-Grained Image Classification, Prateek Shroff et al. route a static feature map through a recurrent attention module over multiple internal time steps. In early iterations, the attention map covers broad, coarse regions; as the cycles progress, the internal attention shifts dynamically toward minute, highly discriminative features (like the specific texture of an animal's fur).
This coarse-to-fine progression is highly effective in sequence generation and mapping tasks. For instance, Kainan Peng et al. show in Non-Autoregressive Neural Text-to-Speech that layer-by-layer attention refinement allows a model to start with a broad positional prior and incrementally sharpen its alignment maps into high-precision text-to-audio trajectories. Similarly, in Stack-Captioning: Coarse-to-Fine Learning for Image Captioning, Jianfei Cai et al. pass attention maps between sequential decoding stages, treating the previous stage's focus as a prior belief to be updated, allowing the model to smoothly transition from global scene context to highly specific local objects.
Co-Evolution and Mutual Refinement
A powerful extension of this paradigm occurs when attention maps and feature representations (or multiple distinct tasks) are updated in a bidirectional, co-evolutionary loop.
In Probabilistic Graph Attention Network With Conditional Kernels for Pixel-Wise Prediction, Dan Xu et al. implement a bidirectional relationship where attention gates and feature representations iteratively refine each other through mean-field inference. The evolving features update the attention gates, and the updated gates filter the features, ensuring mutual optimization.
This mutual refinement is also effective across different modalities or tasks. Yifei Huang demonstrates this in Mutual Context Network for Jointly Estimating Egocentric Gaze and Action by alternating between estimating a spatial attention map and predicting an action label, leveraging the reciprocal relationship between intent and focus until both predictions stabilize. For feature fusion, Yimian Dai shows in Attentional Feature Fusion that using a preliminary fusion step's output as context for a secondary attention weighting stage mitigates the information loss of poor initial integration, leading to far more robust feature selection.
Why It Works: Noise Suppression and Soft Clustering
Why does carrying attention forward outperform standard independent calculations?
First, it acts as a powerful de-noising mechanism. As Chaorui Deng demonstrates in Visual Grounding via Accumulated Attention, accumulating attention weights across multiple rounds of reasoning systematically suppresses background noise and redundancy while amplifying the target signal.
Second, it can be mathematically framed as an optimization or clustering problem. Xia Li proves this in Expectation-Maximization Attention Networks for Semantic Segmentation by reformulating the attention mechanism as an iterative Expectation-Maximization (EM) clustering process. Instead of computing a massive, noisy pixel-to-pixel affinity matrix, the model iteratively updates a compact set of basis vectors. Each internal step refines these bases to better represent the global semantic structure, converging on a clean, representative set of features.
By treating attention as a continuous, refining conversation rather than a series of disconnected, memoryless snapshots, networks achieve sharper boundaries, better noise rejection, and superior parameter efficiency across text, speech, and vision domains.