Implicit Optimization with Non-Zero Mean Data

This direction is merged into another direction: linear attention simplification. Its last version stays readable and citable here.

Version 1 of 110 papers18 conceptsfrozen 2026-10-04

In theoretical machine learning, we often assume that data is "centered"—that its features have a mean of zero. This assumption simplifies the math, but it hides a fundamental mechanism of how neural networks actually learn. Recent research suggests that the non-zero mean of a data distribution is not just a nuisance to be normalized away; rather, it is a "shared signal" that enables models to find sparse solutions and perform sophisticated in-context reasoning.

The Stability-Sparsity Connection

When training a neural network, we want it to find a "rich" solution—one that identifies the most important features (sparsity) rather than just fitting the noise. Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry demonstrated in Implicit Bias of the Step Size in Linear Diagonal Neural Networks that the optimization step size is a primary driver of this sparsity.

The intuition is rooted in optimization stability. If you use a large step size (near the "edge of stability"), the optimizer is forced to keep the weights small to avoid exploding. However, this regularizing effect is mathematically contingent on the data having a non-zero mean.

If the data is perfectly centered at zero, the stability constraints on the optimizer loosen. The mathematical link between the "speed" of the update and the "norm" of the predictor weakens, and the model loses its bias toward sparse, generalizable solutions. This suggests that standard preprocessing, like batch normalization or zero-mean centering, might inadvertently disable the very mechanism that helps large learning rates regularize a model.

Why In-Context Learning Needs a "Prior"

The importance of non-zero means extends beyond training weights to how Large Language Models (LLMs) process information on the fly. A popular theory suggests that In-Context Learning (ICL) is essentially the model performing a form of implicit Gradient Descent (GD) on the examples provided in the prompt.

However, Shifeng Xie, Rui Yuan, and their colleagues point out in The Initialization Determines Whether In-Context Learning Is Gradient Descent that this analogy breaks down in the real world. Most theoretical proofs of "ICL-as-GD" assume zero-mean data. When the data has a bias (a non-zero mean), standard attention mechanisms fail to mimic gradient descent effectively. The performance gap between the model's behavior and actual optimization increases linearly with the square of the data's mean.

To fix this, the model needs an initial guess. By allowing the model to learn a specific initialization for its query—essentially a "prior" that accounts for the non-zero mean—the connection to gradient descent is restored.

The MLP as a Learnable Initialization

If the attention mechanism handles the "gradient update" part of learning, what handles the "initialization"?

Ruiqi Zhang, Jingfeng Wu, and Peter Bartlett argue in In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization that this is the hidden purpose of the Multi-Layer Perceptron (MLP) in a Transformer block.

In their framework:

  1. The MLP learns the "shared signals" across different tasks (the non-zero mean or common baseline).
  2. The Attention layer uses those signals as a starting point to perform one-step gradient descent on the specific task in the prompt.

Without the MLP, the model starts from zero for every new task. With the MLP, the model starts from a "learnable initialization" that captures the inherent bias of the data distribution. This combination allows a single Transformer block to achieve near-optimal performance on biased tasks where simpler, zero-mean-assuming models fail.

Summary of Intuition

The recurring theme across these works is that bias is information.

  • In training, a non-zero mean allows large step sizes to "feel" the scale of the data, forcing the model toward sparse solutions.
  • In inference (ICL), a non-zero mean represents the "common ground" between tasks.

By moving away from the zero-mean assumption, researchers are discovering that neural networks use specific components—like MLPs and large step sizes—to capture and exploit these shared signals, effectively turning "biased data" into a roadmap for better optimization.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_implicit_optimization_with_non_zero_mean_d3d70d2a,
  title = {Implicit Optimization with Non-Zero Mean Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-optimization-with-non-zero-mean-data/rdr_d3d70d2a826a41dcbb690ed06d869631},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_implicit_optimization_with_non_zero_mean_d3d70d2a_v1,
  title = {Implicit Optimization with Non-Zero Mean Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-optimization-with-non-zero-mean-data/rdr_d3d70d2a826a41dcbb690ed06d869631/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}