Attention Sinks and Outliers in Attention Residuals

Haozheng Luo, Haoran Dai, Shaoyang Zhang, Xi Chen, Eric Hanchen Jiang, Yijiang Li, Jingyuan Huang, Chenghao Qiu, Chenwei Xu, Zhenyu Pan, Haotian Zhang, Binghui Wang, Yan ChenarXiv 2026 · DOI

BibTeX
@misc{luo2026attention,
  author = {Haozheng Luo and Haoran Dai and Shaoyang Zhang and Xi Chen and Eric Hanchen Jiang and Yijiang Li and Jingyuan Huang and Chenghao Qiu and Chenwei Xu and Zhenyu Pan and Haotian Zhang and Binghui Wang and Yan Chen},
  title = {{Attention Sinks and Outliers in Attention Residuals}},
  year = {2026},
  doi = {10.48550/arxiv.2605.17887},
}

Loading related content...

Figure