No article text was recorded for this version.
representation over-smoothing
Version 1 of 183 papers100 conceptsfrozen 2026-10-04
Papers in this version
- Blog: Reconstruction Probing
- Blog: Adaptive Bi-directional Attention: Exploring Multi-Granularity Representations for Machine Reading Comprehension
- Blog: Boosting Explainability through Selective Rationalization in Pre-trained Language Models
- Blog: Augmented Shortcuts for Vision Transformers
- Blog: Understanding and Improving Knowledge Distillation for Quantization Aware Training of Large Transformer Encoders
- Blog: AttCAT: Explaining Transformers via Attentive Class Activation Tokens
- Blog: PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensation
- Blog: Channel Propagation Networks for Refreshable Vision Transformer
- Blog: Polynomial-based Self-Attention for Table Representation Learning
- Blog: Revisiting Over-smoothing in BERT from the Perspective of Graph
- Blog: Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice
- Blog: Boosting Explainability through Selective Rationalization in Pre-trained Language Models
- Blog: Transformer Meets Twicing: Harnessing Unattended Residual Information
- Blog: Transformers need glasses! Information over-squashing in language tasks
- Blog: You Do Not Fully Utilize Transformer's Representation Capacity
- Blog: Why do LLMs attend to the first token?
- Blog: SIBO: A Simple Booster for Parameter-Efficient Fine-Tuning
- Blog: Multi-Criteria Token Fusion with One-Step-Ahead Attention for Efficient Vision Transformers
- Blog: Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation
- Blog: Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN
- Blog: Value Residual Learning
- Blog: ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition
- Blog: Mitigating Undisciplined Over-Smoothing in Transformer for Weakly Supervised Semantic Segmentation
- Blog: Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning
- Blog: Consensus Is All You Get: The Role of Attention in Transformers
- Blog: Attention Projection Mixing with Exogenous Anchors
- Blog: Exploiting Sentence-Level Representations for Passage Ranking
- Blog: Elliptical Attention
- Blog: More Expressive Attention with Negative Weights
- Blog: LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning
- Blog: Scaled ReLU Matters for Training Vision Transformers
- Blog: Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
- Blog: Distinguishability Calibration to In-Context Learning
- Blog: PIDformer: Transformer Meets Control Theory
- Blog: Zero-guidance Segmentation Using Zero Segment Labels
- Blog: Pre-trained Vision and Language Transformers are Few-Shot Incremental Learners
- Blog: Researches Advanced in the Development and Application of Transformers
- Blog: Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning
- Blog: Scaled ReLU Matters for Training Vision Transformers
- Blog: Long-Context Generalization with Sparse Attention
- Blog: Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations
- Blog: The Mean-Field Dynamics of Transformers
- Blog: Information Extraction Network Based on Multi-Granularity Attention and Multi-Scale Self-Learning
- Blog: SAM-DCE: Addressing Token Uniformity and Semantic Over-Smoothing in Medical Segmentation
- Blog: MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scores
- Blog: Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Transformers
- Blog: Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning.
- Blog: PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
- Blog: Revisiting Kernel Attention with Correlated Gaussian Process Representation
- Blog: MultiMax: Sparse and Multi-Modal Attention Learning
- Blog: Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals
- Blog: Why do LLMs attend to the first token?
- Blog: Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
- Blog: Exploiting Sentence-Level Representations for Passage Ranking
- Blog: Alleviating Over-smoothing for Unsupervised Sentence Representation
- Blog: Exploiting the Social-Like Prior in Transformer for Visual Reasoning
- Blog: Token Contrast for Weakly-Supervised Semantic Segmentation
- Blog: Wavy Transformer
- Blog: Probing the Embedding Space of Transformers via Minimal Token Perturbations
- Blog: An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention
Showing 60 of 83 papers.
History & citation
Current · Permanent link
- Preserving Token Diversity in Deep Transformers merged into this direction (51% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_representation_over_smoothing_f5204760,
title = {representation over-smoothing},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/representation-over-smoothing/rdr_f5204760c3b74c6ebe4610742ce251f6},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}This exact version (version 1, frozen text):
@misc{lacuna_representation_over_smoothing_f5204760_v1,
title = {representation over-smoothing},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/representation-over-smoothing/rdr_f5204760c3b74c6ebe4610742ce251f6/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}