Learning and Estimation Under Model Misspecification

126 concepts117 papers2 proposals

In classical statistical learning theory, we often treat the world as a tidy place. We assume realizability: the true data-generating distribution $P^*$ lives comfortably within our chosen model class $\mathcal{M}$. Under this well-specified assumption, maximum likelihood estimators converge cleanly to the true parameters, and optimization landscapes behave predictably.

But what happens when the model is wrong? In practice, models are almost always misspecified; the true distribution $P^*$ lies outside $\mathcal{M}$. This shift from a well-specified to an agnostic or misspecified setting fundamentally alters how algorithms learn, optimize, and generalize. Instead of finding the absolute truth, the model must find the best possible approximation within its limited capacity—a process fraught with unexpected mathematical traps and phase transitions.

When More Data Breeds Confidence in Error

Intuitively, one might expect that collecting more data always improves a model's performance, or at least causes it to plateau. However, under model misspecification, this intuition breaks down spectacularly.

Consider the dipping phenomenon characterized by Duin et al. (2012). When a classifier's hypothesis space cannot represent the true data distribution, its performance can peak at a finite training set size and actively worsen as the number of samples approaches infinity. As the model receives more data, it becomes highly confident in its incorrect parameters, converging toward a stable but globally inferior solution within its restricted hypothesis space.

Performance
    ^
    |       /\
    |      /  \
    |     /    \_______  <- Misspecified Model (Dipping)
    |    /
    |   /______________  <- Well-specified Model
    +---------------------> Data Volume

Misspecification also breaks the optimization tools we rely on to navigate loss landscapes. For instance, natural gradient descent often uses the "empirical Fisher" matrix as a computationally cheap proxy for the true Hessian. However, Kunstner et al. (2019) demonstrated that this approximation relies entirely on the model being correctly specified. When the model is misspecified, the empirical Fisher can be arbitrarily wrong—even near a minimum—distorting optimization paths and causing pathological training behavior.

The Information-Theoretic Projection

If a misspecified model cannot find the true distribution, what does it find? The answer lies in information-theoretic projections. When $P^* \notin \mathcal{M}$, Maximum Likelihood Estimation (MLE) acts as a geometric projection, minimizing the Kullback-Leibler (KL) divergence between the true distribution and the model class.

For complex sequential data, Moulines et al. (2012) proved that even when a Hidden Markov Model (HMM) is misspecified, the MLE remains consistent in a specific sense: it converges robustly to the parameter that minimizes this information-theoretic distance. This works because the underlying processes "forget" their initial conditions exponentially fast, stabilizing the projection over time.

However, this projection behaves differently depending on the estimator used. Grünwald et al. (2005) showed that in-model estimators (like standard MLE) and out-of-model estimators (which aggregate multiple models) exhibit fundamentally different asymptotic risks under misspecification. Similarly, in information-theoretic model selection, the authors of Srebro et al. (2025) showed that the Minimum Description Length (MDL) principle can suffer from persistent under-regularization and overfitting when the true distribution is absent from the candidate set, requiring a careful re-evaluation of standard selection rules.

Designing for Agnostic Regimes

To build algorithms that survive real-world misspecification, researchers have shifted toward agnostic learning frameworks, where guarantees are made relative to the best possible hypothesis in the class, rather than an absolute truth.

For Empirical Risk Minimization (ERM), Hanneke et al. (2025) mapped out a complete theoretical "trichotomy" of learning rates in agnostic settings, showing how combinatorial properties of the concept class dictate whether learning slows down or remains exponentially fast under misspecification. To handle distribution mismatch in unsupervised tasks, Afzali et al. (2025) introduced algorithms for private density estimation of Gaussian Mixture Models (GMMs) that maintain sample-complexity bounds even when the true data distribution deviates from a clean mixture.

Another powerful strategy is to change what we optimize. Traditional likelihood-based methods degrade quickly under structural misspecification because they try to model the entire joint distribution. To circumvent this, Domke (2013) proposed using marginal-based loss functions for graphical models. By directly optimizing the accuracy of predicted marginals through a fixed number of approximate inference iterations, the model learns parameters optimized for the actual downstream task, bypassing the fragility of global likelihood matching.

Identifiability and Architecture Collapse

In modern over-parameterized architectures like Mixture of Experts (MoE) or deep neural networks, misspecification introduces unique structural bottlenecks.

In MoE models, which route inputs to different specialized sub-networks via a gating function, parameter estimation becomes highly sensitive to distribution mismatch. Nguyen et al. (2023) and Nguyen et al. (2024) explored this phenomenon, showing that standard softmax gating functions cause a "parameter collapse" when the model is misspecified, severely slowing down convergence rates because the gating weights fail to distinguish between experts cleanly.

To achieve robust generalization despite these architectural mismatches, we must rethink what the network is actually learning. Xu et al. (2021) demonstrated that while over-parameterization and misspecification make it impossible to recover the true parameters of a data-generating function, we can still uniquely identify the true underlying function itself. By shifting the theoretical focus from parameter recovery to function identification, they proved that networks can achieve robust generalization across arbitrary distribution shifts, even when the model architecture is fundamentally misspecified.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_model_misspecification_robustness_to_mod_a33c58a6,
  title = {model misspecification (robustness to model misspecification)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-misspecification-robustness-to-model-misspecification/rdr_a33c58a6873d48528e01d8b2d9719274},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_model_misspecification_robustness_to_mod_a33c58a6_v1,
  title = {model misspecification (robustness to model misspecification)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-misspecification-robustness-to-model-misspecification/rdr_a33c58a6873d48528e01d8b2d9719274/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake