No article text was recorded for this version.
attention head redundancy (attention head overlap)
Version 1 of 129 papers33 conceptsfrozen 2026-10-04
Papers in this version
- Blog: UniForm: A Reuse Attention Mechanism for Efficient Transformers on Resource-Constrained Edge Devices
- Blog: Cascaded Head-colliding Attention
- Blog: Quaternion Self-Attention with Shared Scores
- Blog: ThinK: Thinner Key Cache by Query-Driven Pruning
- Blog: Libra: Building Decoupled Vision System on Large Language Models
- Blog: HoRA: Cross-Head Low-Rank Adaptation with Joint Hypernetworks
- Blog: Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA
- Blog: Graph Reasoning Transformer for Image Parsing
- Blog: Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models
- Blog: Orthogonality Constrained Multi-Head Attention For Keyword Spotting
- Blog: FourierFormer: Transformer Meets Generalized Fourier Integral Theorem
- Blog: Fast-StrucTexT: An Efficient Hourglass Transformer with Modality-guided Dynamic Token Merge for Document Understanding
- Blog: LeanK: Learnable K Cache Channel Pruning for Efficient Decoding
- Blog: EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention
- Blog: Continuous Decomposition of Granularity for Neural Paraphrase Generation
- Blog: Improving Transformers with Probabilistic Attention Keys
- Blog: Serial Low-rank Adaptation of Vision Transformer
- Blog: MicroViT: A Vision Transformer with Low Complexity Self Attention for Edge Device
- Blog: Attention Mixtures for Time-Aware Sequential Recommendation
- Blog: Pyramid Attention Network for Medical Image Registration
- Blog: Improving Transformers with Probabilistic Attention Keys
- Blog: SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design
- Blog: A Dynamic Head Importance Computation Mechanism for Neural Machine Translation
- Blog: S2AFormer: Strip Self-Attention for Efficient Vision Transformer
- Blog: Transformers with Multiresolution Attention Heads
- Blog: A Tensorized Transformer for Language Modeling
- Blog: Finding the Pillars of Strength for Multi-Head Attention
- Blog: IFORMER: INTEGRATING CONVNET AND TRANSFORMER FOR MOBILE APPLICATION
- Blog: Implicit Kernel Attention
History & citation
Current · Permanent link
- This direction split from Eliminating Redundancy in Multi-Head Attention (66% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_attention_head_redundancy_attention_head_ec36656a,
title = {attention head redundancy (attention head overlap)},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/attention-head-redundancy-attention-head-overlap/rdr_ec36656a1874491f89bd816b986b352f},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}This exact version (version 1, frozen text):
@misc{lacuna_attention_head_redundancy_attention_head_ec36656a_v1,
title = {attention head redundancy (attention head overlap)},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/attention-head-redundancy-attention-head-overlap/rdr_ec36656a1874491f89bd816b986b352f/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}