In the standard Transformer architecture, the "multi-head" in multi-head attention (MHA) is often a misnomer for diversity. While we might hope different heads learn distinct features, they often suffer from "uniformity redundancy," where multiple heads focus on the same easy-to-process regions. Furthermore, standard MHA treats every token relationship with the same global or fixed-window lens, leading to a "BERT tax"—a quadratic computational cost that makes processing long sequences like DNA or high-resolution images prohibitively expensive.
A growing body of research suggests a more efficient path: Heterogeneous Attention. Instead of forcing every head to look at the data through the same window, we can partition heads to operate at different scales, resolutions, and receptive fields simultaneously.
The Hydra Intuition: One Layer, Many Lenses
The core idea is to treat a single Transformer layer like a "Hydra." As Ali Hassani and Steven Walton 0001 describe in their work on StyleNAT, a model can assign different receptive field sizes to individual heads within the same layer.
Imagine a photographer with a camera that has eight lenses firing at once. Some lenses are macro, capturing fine-grained textures; others are wide-angle, capturing the global composition. By allowing heads to "specialize" in different spatial scales, the model doesn't have to choose between local precision and global context—it gets both in a single pass.
Mechanisms of Heterogeneity
Researchers have identified several ways to "shunt" or diversify these attention heads:
1. Variadic Window Sizes
The most direct approach is to assign different window sizes to different heads. In the Multi-Window Masked Autoencoder (MW-MAE), Sarthak Yadav demonstrates that audio signals, which are inherently multi-scale, are better captured when some heads use small, local windows while others use large, global ones. Similarly, Qirong Yang uses this strategy in TrinityDNA to capture both short-range genetic motifs and long-range evolutionary dependencies without the entropy collapse typically seen in long-sequence modeling.
2. Multi-Scale Token Aggregation
Rather than just changing the window size, the Shunted Transformer by Sucheng Ren varies the granularity of the tokens themselves. Some heads receive high-resolution tokens to identify small objects, while other heads receive downsampled, "coarse-grained" tokens to represent large structures. This reduces the total number of key-value pairs the model must process, slashing computational costs while maintaining high accuracy.
3. Frequency and Dilation Splitting
In vision tasks, Zizheng Pan introduced HiLo attention, which explicitly splits heads into high-frequency (local) and low-frequency (global) processors. Other models, like the image compression framework by Zhengxin Chen, use varying dilation rates across heads. This allows the model to "skip" pixels in some heads to see a wider area without increasing the number of parameters.
Why It Works: Breaking the Low-Pass Filter
Standard self-attention often acts as a low-pass filter, smoothing out features across a sequence. While this helps with general context, it can erase sharp, nuanced differences (heterophily).
Tan Minh Nguyen addresses this with the $p$-Laplacian Transformer, which assigns different regularization strengths to different heads. Some heads are tuned to capture smooth similarities, while others are "sharpened" to detect abrupt changes. This heterogeneity allows the model to adapt to diverse data types where the relationship between tokens isn't always one of simple similarity.
Efficiency Without Sacrifice
The primary benefit of heterogeneous heads is the elimination of the "all-or-nothing" trade-off in attention. In traditional models, if you want global context, you pay the $O(N^2)$ price for every head.
By partitioning the heads:
- Local heads maintain high-resolution detail with minimal cost.
- Global/Dilated heads provide context using a fraction of the tokens.
- Positional scaling, as seen in the work of Zhenyu Zhang, allows models to "find" information in the middle of long documents by rescaling positional indices differently for each head, effectively stretching the model's focus.
This architectural shift, seen in models ranging from Jiezhong Qiu’s Blockwise Self-Attention to Pascal Notin’s Tranception, suggests that the future of Transformers lies not in making models larger, but in making their attention heads more diverse.