The central promise of multi-head attention is that a model can simultaneously attend to information from different representation subspaces at different positions. In theory, one head might focus on syntax, another on local visual textures, and a third on global semantic context. However, a recurring finding across both Natural Language Processing and Computer Vision is that these heads often "collapse" into redundant patterns, focusing on the same most salient features and wasting the model's expressive capacity.
The Salience Trap and Attention Collapse
The fundamental problem, often termed "attention collapse" or "representation collapse," occurs because standard training objectives—like cross-entropy or contrastive loss—are "greedy." They encourage the model to identify the most discriminative signal as quickly as possible. As Zhouhan Lin and colleagues noted in A Structured Self-Attentive Sentence Embedding, without a mechanism to enforce diversity, multiple attention components will naturally gravitate toward the same salient features, such as a single dominant word in a sentence or a bright object in an image.
This redundancy is particularly acute in deep layers. Zhenda Xie and Han Hu demonstrated in Revealing the Dark Secrets of Masked Image Modeling that while Masked Image Modeling (MIM) maintains head diversity, supervised and contrastive models often see their attention maps become overly homogeneous in deeper layers. This homogeneity prevents the model from capturing the fine-grained, local details necessary for complex tasks like image segmentation or deepfake detection, where the "devil" is often in subtle, non-salient artifacts.
Forcing Diversity through Repulsion and Orthogonality
To combat this, researchers have introduced explicit "diversity pressures" during training. One intuitive approach is to treat attention heads as physical particles that should repel one another. In Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference, Bang An proposes a "repulsive force" derived from kernelized particle optimization. By adding this force, the model is mathematically discouraged from allowing heads to converge on identical parameters, effectively pushing them to explore different "modes" of the input data.
A more geometric approach involves orthogonality constraints. If the weight matrices or the resulting attention affinity maps are forced to be orthogonal, they are by definition non-redundant. Aston Zhang and collaborators showed in On Orthogonality Constraints for Transformers that encouraging these matrices to be orthogonal improves numerical stability and prevents the "rank collapse" of the attention mechanism, leading to better generalization in machine translation. Similarly, Po-Yao Huang utilized a "diversity loss" in Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations to ensure that different heads ground themselves in different visual objects within a scene.
Architectural Inductive Biases for Variety
Beyond loss functions, the way we structure the attention mechanism itself can dictate its diversity. Traditional multi-head attention splits the embedding dimension into equal subspaces. However, Dayu Li and Jinshan Pan argue in Devil is in the Uniformity that this uniformity is a bottleneck. By using hierarchical subspace splitting and cross-layer caches, they force heads to become "diverse learners," which is essential for image restoration tasks where both coarse and fine details must be recovered.
In the realm of text, Jinhua Du and Andy Way moved from 1-D attention vectors to 2-D matrices in Multi-Level Structured Self-Attentions for Distantly Supervised Relation Extraction. This "structured" approach allows the model to capture multiple semantic "aspects" of a sentence simultaneously, ensuring that the representation of a sentence isn't just a single average, but a collection of distinct viewpoints.
Theoretical Limits and Scaling
The drive for diversity is not just an empirical heuristic; it has deep theoretical roots. Blake Bordelon and Hamza Tahir Chaudhry explored the Infinite Limits of Multi-head Transformer Dynamics, discovering that standard width-scaling (increasing the size of each head) actually causes multi-head attention to collapse into single-head behavior. To preserve functional diversity, they argue that one must scale the number of heads rather than their internal dimensions.
This suggests that the "multi-head" nature of Transformers is a fragile property. Whether through Bayesian repulsion, orthogonality penalties, or careful architectural choices, the goal remains the same: preventing the model from taking the easy path of redundancy and forcing it to utilize its full capacity to see the world from multiple, non-overlapping perspectives.