No article text was recorded for this version.
linear attention simplification
Version 2 of 2414 papers489 conceptsfrozen 2026-10-04
Papers in this version
- Blog: MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map
- Blog: Transformers May Learn to Classify In-Context by Context-Adaptive Kernel Gradient Descent
- Blog: Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective
- Blog: Training Dynamics of In-Context Learning in Linear Attention
- Blog: A phase transition between positional and semantic learning in a solvable model of dot-product attention<sup>*</sup>
- Blog: The Closeness of In-Context Learning and Weight Shifting for Softmax Regression
- Blog: Asymptotic Study of In-context Learning with Random Transformers through Equivalent Models
- Blog: First De-Trend then Attend: Rethinking Attention for Time-Series Forecasting
- Blog: The Closeness of In-Context Learning and Weight Shifting for Softmax Regression
- Blog: Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models
- Blog: Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context
- Blog: Graph Attention is Not Always Beneficial: A Theoretical Analysis of Graph Attention Mechanisms via Contextual Stochastic Block Models
- Blog: Provable In-Context Learning of Linear Systems and Linear Elliptic PDEs with Transformers
- Blog: Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape
- Blog: Transformers as Multi-Task Feature Selectors: Generalization Analysis of In-Context Learning
- Blog: A Solvable Attention for Neural Scaling Laws
- Blog: Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time
- Blog: Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens
- Blog: Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
- Blog: The Initialization Determines Whether In-Context Learning Is Gradient Descent
- Blog: Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time
- Blog: Transformers Learn Nonlinear Features In Context
- Blog: How do Transformers Perform In-Context Autoregressive Learning ?
- Blog: The Role of Logic and Automata in Understanding Transformers
- Blog: Gating is Weighting: Understanding Gated Linear Attention through In-context Learning
- Blog: Attention Scheme Inspired Softmax Regression
- Blog: Optimal Attention Temperature Enhances In-Context Learning under Distribution Shift
- Blog: On the Training Convergence of Transformers for In-Context Classification
- Blog: Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs
- Blog: Finite-Time Analysis of Gradient Descent for Shallow Transformers
- Blog: Probability Distributions Computed by Hard-Attention Transformers
- Blog: How Well Can Transformers Emulate In-Context Newton's Method?
- Blog: Infinite attention: NNGP and NTK for deep attention networks
- Blog: Logical Languages Accepted by Transformer Encoders with Hard Attention
- Blog: On the Role of Unstructured Training Data in Transformers' In-Context Learning Capabilities
- Blog: Masked Hard-Attention Transformers and Boolean RASP Recognize Exactly the Star-Free Languages
- Blog: How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?
- Blog: Attention-based clustering
- Blog: Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective
- Blog: Linear attention is (maybe) all you need (to understand Transformer optimization)
- Blog: In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
- Blog: Approximation Bounds for Transformer Networks with Application to Regression
- Blog: On learning linear dynamical systems in context with attention layers
- Blog: Trained Transformers Learn Linear Models In-Context
- Blog: Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers
- Blog: How do Transformers perform In-Context Autoregressive Learning?
- Blog: Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality (extended abstract)
- Blog: Towards Provable Emergence of In-Context Reinforcement Learning
- Blog: Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmetic
- Blog: Decomposing Attention To Find Context-Sensitive Neurons
- Blog: In-Context Learning behaves as a greedy layer-wise gradient descent algorithm
- Blog: Saturated Transformers are Constant-Depth Threshold Circuits
- Blog: In-Context Learning by Linear Attention: Exact Asymptotics and Experiments
- Blog: Training Dynamics of In-Context Learning in Linear Attention
- Blog: Limits to Depth Efficiencies of Self-Attention
- Blog: On the Robustness of Transformers against Context Hijacking for Linear Classification
- Blog: Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective
- Blog: Linear attention is (maybe) all you need (to understand transformer optimization)
- Blog: Asymptotic theory of in-context learning by linear attention
- Blog: The Closeness of In-Context Learning and Weight Shifting for Softmax Regression
Showing 60 of 414 papers.
History & citation
Current · Permanent link
- Logical Characterization of Transformer Expressivity merged into this direction (57% overlap)
- Formal Expressivity and Symbolic Logic in Transformers merged into this direction (69% overlap)
- In-Context Learning as Gradient Descent merged into this direction (67% overlap)
- Implicit Optimization with Non-Zero Mean Data merged into this direction (50% overlap)
- Dynamic Metastability in Transformer Self Attention merged into this direction (56% overlap)
- gradient descent equivalence split from this direction (69% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_linear_attention_simplification_797bd50d,
title = {linear attention simplification},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/linear-attention-simplification/rdr_797bd50dbae04561a3771e45e52de634},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}This exact version (version 2, frozen text):
@misc{lacuna_linear_attention_simplification_797bd50d_v2,
title = {linear attention simplification},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/linear-attention-simplification/rdr_797bd50dbae04561a3771e45e52de634/v/2},
note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}