Agnostic Reinforcement Learning and Model Misspecification

This direction is merged into another direction: Bellman completeness. Its last version stays readable and citable here.

Version 1 of 153 papers56 conceptsfrozen 2026-10-04

Traditional reinforcement learning (RL) theory is built on a comforting but often fragile assumption: realizability. This is the belief that the "truth"—whether it is the environment’s transition physics, the optimal value function, or an expert’s policy—resides perfectly within the learner's chosen hypothesis class.

In the real world, this assumption is almost always violated. A neural network is a simplified approximation of a complex physical system; a linear model cannot capture the nuances of human behavior. When the model is "misspecified," standard algorithms often collapse. Agnostic learning is the research direction dedicated to breaking this dependency, seeking algorithms that perform as well as the best possible candidate in a given class, even when that class is inherently flawed.

The Realizability Trap

Most RL guarantees rely on the idea that if we see enough data, we will eventually find the "true" model. As Gene Li argues in Agnostic Reinforcement Learning: Foundations and Algorithms, this creates a massive gap between theory and practice. If the true environment is not in our model class, a learner might become overconfident in a wrong model, leading to catastrophic errors in exploration.

The challenge of model misspecification is particularly acute in value-based learning. Yunfan Li notes in On the Model-Misspecification in Reinforcement Learning that even "locally-bounded" errors—where the model is only slightly off in certain states—can compound over time. To solve this, researchers have moved toward "agnostic" frameworks that don't ask "Is this model true?" but rather "How close is this model to the best we can do?"

Knowing What You Don't Know

One of the foundational pillars of this direction is the KWIK (Knows What It Knows) framework. In Agnostic KWIK learning and efficient approximate reinforcement learning, István Szita and Csaba Szepesvari proved that an agent can learn efficiently if it can identify when a state is "unfamiliar" or "unrepresentable."

Instead of trying to fit a square peg in a round hole, a KWIK-learner admits when its hypothesis class is insufficient. This allows the agent to maintain a "close enough" model of the environment, ensuring that even with a misspecified model, the resulting policy's performance remains bounded relative to the optimal behavior.

Agnostic Imitation: Learning from "Better" Experts

Imitation learning provides a clear window into the agnostic problem. Usually, we assume the learner can perfectly mimic the expert. But what if the expert is a human with a complex internal state, and the learner is a simple linear controller?

Yichen Li and Chicheng Zhang address this in Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms. They demonstrate that standard algorithms like DAgger can fail when the expert's policy is outside the learner's hypothesis space. Their solution, BOOTSTRAP-DAGGER, uses ensembles to quantify the "complexity gap" between the expert and the learner, allowing the agent to find the best possible approximation without requiring perfect mimicry.

Similarly, Dylan J Foster and Adam Block show in Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning that by switching to a logarithmic loss, offline learners can become surprisingly robust to misspecification, matching the performance of interactive learners even when the model is too simple to capture the expert's full logic.

The Geometry of Tractability

Why is agnostic learning harder than realizable learning? Zeyu Jia, Alexander Rakhlin, and Nati Srebro pinpoint the answer in the structural geometry of the policy class. In When is Agnostic Reinforcement Learning Statistically Tractable?, they introduce the concept of spanning capacity.

In a realizable world, you only need to find the one "true" point in your hypothesis space. In an agnostic world, you must explore the entire "span" of the space to ensure you haven't missed a better-performing (though still imperfect) policy. This explains why some problems are easy to solve with a simulator but become exponentially difficult when the agent has to learn through direct, "agnostic" interaction with the world.

Pessimism as a Defense

In the "offline" setting—where we learn from a fixed dataset—misspecification is even more dangerous. If the data doesn't cover the whole state space, a misspecified model might hallucinate high rewards in unexplored regions.

To counter this, Masatoshi Uehara and Wen Sun advocate for pessimism. In Pessimistic Model-based Offline RL: PAC Bounds and Posterior Sampling under Partial Coverage, they show that by being "pessimistic" about what the model doesn't know, an agent can achieve strong performance guarantees even when the model is misspecified and the data coverage is poor. This shift from "finding the truth" to "avoiding the unknown" is the hallmark of the agnostic approach.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_agnostic_reinforcement_learning_and_mode_a356e682,
  title = {Agnostic Reinforcement Learning and Model Misspecification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/agnostic-reinforcement-learning-and-model-misspecification/rdr_a356e682124549da9fb51f8644f80b7a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_agnostic_reinforcement_learning_and_mode_a356e682_v1,
  title = {Agnostic Reinforcement Learning and Model Misspecification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/agnostic-reinforcement-learning-and-model-misspecification/rdr_a356e682124549da9fb51f8644f80b7a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}