No article text was recorded for this version.
attention head redundancy (attention mechanism redundancy)
Version 1 of 1117 papers144 conceptsfrozen 2026-10-04
Papers in this version
- Blog: Revealing the Dark Secrets of BERT
- Blog: Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks
- Blog: On Attention Redundancy: A Comprehensive Study
- Blog: Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads
- Blog: Efficient Transformer-Based Abstractive Urdu Text Summarization Through Selective Attention Pruning
- Blog: Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- Blog: On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
- Blog: Picking the Underused Heads: A Network Pruning Perspective of Attention Head Selection for Fusing Dialogue Coreference Information
- Blog: Do Multilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads?
- Blog: Are Sixteen Heads Really Better than One?
- Blog: Role of Bias Terms in Dot-Product Attention
- Blog: Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection
- Blog: T3-Vis: visual analytic for Training and fine-Tuning Transformers in NLP
- Blog: Learning When to Attend: Conditional Memory Access for Long-Context LLMs
- Blog: Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models
- Blog: Understanding Self-Attention of Self-Supervised Audio Transformers
- Blog: Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- Blog: LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
- Blog: Understanding Multi-Head Attention in Abstractive Summarization
- Blog: Differentiable Subset Pruning of Transformer Heads
- Blog: Chop Chop BERT: Visual Question Answering by Chopping VisualBERT's Heads
- Blog: A Primer in BERTology: What we know about how BERT works
- Blog: Are Sixteen Heads Really Better than One?
- Blog: $W_K, W_V$ is Probably All You Need: On the Necessity of the Query, Key, and Value Weight Triplet in Self-Attention Transformers
- Blog: Towards Redundancy Reduction in Diffusion Models for Efficient Video Super-Resolution
- Blog: Effective Model Pruning
- Blog: Memory-efficient Transformers via Top-k Attention
- Blog: Interpretable Neural Predictions with Differentiable Binary Variables
- Blog: Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference
- Blog: ThinK: Thinner Key Cache by Query-Driven Pruning
- Blog: ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching
- Blog: Losing Heads in the Lottery: Pruning Transformer Attention in Neural Machine Translation
- Blog: A Closer Look at Transformer Attention for Multilingual Translation
- Blog: A Mixture of h-1 Heads is Better than h Heads
- Blog: DEFA: Efficient Deformable Attention Acceleration via Pruning-Assisted Grid-Sampling and Multi-Scale Parallel Processing
- Blog: Opinion Mining with Deep Contextualized Embeddings
- Blog: Reverse-Engineering the Retrieval Process in GenIR Models
- Blog: Understanding Video Transformers via Universal Concept Discovery
- Blog: Exploring Sparsity in Graph Transformers
- Blog: SparQ Attention: Bandwidth-Efficient LLM Inference
- Blog: Self-Attention for Incomplete Utterance Rewriting
- Blog: On the weak link between importance and prunability of attention heads
- Blog: Alleviating the Inequality of Attention Heads for Neural Machine Translation
- Blog: How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers
- Blog: Divergent Token Metrics: Measuring degradation to prune away LLM components - and optimize quantization
- Blog: Entropy-Based Block Pruning for Efficient Large Language Models
- Blog: The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval
- Blog: Identifying and Evaluating Inactive Heads in Pretrained LLMs
- Blog: AutoFormer: Searching Transformers for Visual Recognition
- Blog: Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
- Blog: LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models
- Blog: RAR: Reversing Visual Attention Re-Sinking for Unlocking Potential in Multimodal Large Language Models
- Blog: Recall, Expand, and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing
- Blog: A Survey of Techniques for Optimizing Transformer Inference
- Blog: Rethinking Low-Rank Adaptation in Vision: Exploring Head-Level Responsiveness Across Diverse Tasks
- Blog: Adaptive Computation Pruning for the Forgetting Transformer
- Blog: QuoKA: Query-Oriented KV Selection for Efficient LLM Prefill
- Blog: Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
- Blog: Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation
- Blog: Masked Spiking Transformer
Showing 60 of 117 papers.
History & citation
Current · Permanent link
- Structural Redundancy in Transformer Attention Mechanisms merged into this direction (68% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_attention_head_redundancy_attention_mech_456813e0,
title = {attention head redundancy (attention mechanism redundancy)},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/attention-head-redundancy-attention-mechanism-redundancy/rdr_456813e0011344acab3773f223de9a4d},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}This exact version (version 1, frozen text):
@misc{lacuna_attention_head_redundancy_attention_mech_456813e0_v1,
title = {attention head redundancy (attention mechanism redundancy)},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/attention-head-redundancy-attention-mechanism-redundancy/rdr_456813e0011344acab3773f223de9a4d/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}