Aligning Predictive Models with Control Objectives

119 concepts83 papers1 proposal

Imagine building a flight simulator to train pilots. To make it realistic, you might spend millions of dollars ensuring that the simulator perfectly models the movement of clouds, the rustle of leaves on the runway, and the exact shade of the sunset. Yet, if the simulator fails to accurately capture the aerodynamic stall behavior of the aircraft wings during a storm, the pilot will learn a catastrophic policy.

This scenario illustrates the objective mismatch problem in reinforcement learning (RL). In both model-based and model-free RL, we routinely break the global goal—maximizing cumulative reward—into smaller, mathematically tractable sub-problems. We train world models to predict the next state, and we train critics to minimize value estimation errors.

The paradox is that an agent can possess a highly accurate world model or a perfectly minimized value error, yet still fail miserably at the actual task. This happens because standard statistical proxies (like Maximum Likelihood Estimation or Mean Squared Error) treat all errors equally. In control, however, some errors are trivial, while others are fatal.

The Prediction-Control Gap in Model-Based RL

In model-based reinforcement learning (MBRL), the agent learns a transition model (a "world model") of the environment to plan future actions. Traditionally, this model is trained using Maximum Likelihood Estimation (MLE) to minimize global state-prediction error.

The objective mismatch arises because a model optimized for global statistical likelihood does not inherently prioritize the specific accuracy required for effective planning. If an environment contains complex but reward-irrelevant dynamics (such as background video noise), an MLE-optimized model will waste its capacity trying to predict that noise rather than focusing on the sparse transitions that trigger rewards.

To bridge this gap, researchers developed decision-aware model learning. Instead of asking "What model best fits the historical data?", they ask "What model helps the policy make the best decisions?"

Standard MBRL (Task-Agnostic):
[Data] ───> Minimize Prediction Error (MLE) ───> World Model ───> Policy Planning

Decision-Aware MBRL:
[Data] ───> Minimize Value/Gradient Error ──────> World Model ───> Policy Planning
                 ▲
                 └─────── Weighted by Task Relevance

Farahmand et al. formalized this insight with Value-Aware Model Learning (VAML) in Value-Aware Loss Function for Model-based Reinforcement Learning. They introduced a loss function that penalizes prediction errors proportionally to how much they impact the downstream value function. Farahmand later expanded this into an iterative framework, Iterative Value-Aware Model Learning, demonstrating that coupling model fitting directly with the policy's current value landscape prevents the model from wasting capacity on task-irrelevant environmental details.

The Gradient Mismatch

A particularly acute form of objective mismatch occurs in policy gradient methods. When an agent uses a world model to optimize a policy, it backpropagates through the model's dynamics to compute policy gradients. This means the policy doesn't just need accurate state predictions; it needs accurate derivatives.

As Li et al. demonstrated in Gradient Information Matters in Policy Optimization by Back-propagating through Model, a dynamics model can achieve exceptionally low state prediction error while simultaneously yielding highly inaccurate gradient directions. When the policy updates along these corrupted gradients, performance stalls.

To fix this, frameworks like Gradient-Aware Model-Based Policy Search (GAMPS), proposed by D'Oro et al. in Gradient-Aware Model-Based Policy Search, explicitly weight the model-learning objective based on the score of the policy gradient. By aligning the training objective with the derivative requirements of the correction step, the model remains highly accurate in the exact directions the policy intends to move.

Evolving Policies and Compounding Errors

The mismatch is further compounded because the agent's policy is a moving target. If a world model is trained on a uniform mixture of all past experiences, it will be highly accurate on average but suffer from high prediction error in the specific states currently visited by the evolving policy.

Wang addressed this in Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy by dynamically reweighting historical data so the model focuses on the agent's current behavior, mitigating compounding rollout errors. Taking this to its logical conclusion, Eysenbach et al. proposed a fully unified framework in Mismatched No More: Joint Model-Policy Optimization for Model-Based RL. By deriving a single global lower bound on expected returns, they optimize both the model and the policy under a shared objective, ensuring that the model's representational budget is inherently focused on transitions critical to task success.

The Actor-Critic Disconnect

The objective mismatch problem is not exclusive to model-based learning; it also fundamentally limits model-free actor-critic architectures.

In a standard actor-critic setup, the two components have distinct mathematical goals:

  1. The Critic minimizes value estimation error (e.g., Temporal Difference error or Bellman residuals).
  2. The Actor updates its policy parameters to maximize the values reported by the critic.

This division of labor introduces a subtle but damaging misalignment.

The Bellman Residual as a Bad Proxy

We often use the Bellman residual as a mathematical shortcut because it is easy to compute and optimize. However, Geist et al. proved in Is the Bellman residual a bad proxy? that minimizing the Bellman residual lacks a guaranteed correlation with actual policy performance. An agent can successfully drive its training error to zero while failing to improve its decision-making quality, because the proxy objective does not preserve the true policy ranking.

Policy Lag and Synchronization Gaps

Because the actor and critic are updated asynchronously, a synchronization gap emerges. The actor network updates its policy based on value estimates that the critic network is still learning.

Interestingly, the critic's internal knowledge of the environment's reward structure is often significantly more advanced than the actor's current policy. As Ki notes in Actor-Critic without Actor, this policy lag creates a training bottleneck where the actor struggles to accurately parameterize the optimal action distributions already understood by the critic.

To resolve this fundamental misalignment, Vaswani et al. introduced a "decision-aware" actor-critic framework in Decision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees. Instead of training the critic purely to minimize value estimation error, they derive a joint lower bound on policy improvement. This explicitly optimizes the critic to help the actor make better policy updates, providing theoretical guarantees for monotonic improvement even when using deep function approximation.

Why Surrogate Objectives Diverge

To stabilize training in deep policy optimization (such as PPO), algorithms rely on surrogate objectives—local mathematical approximations of the reward landscape that prevent the policy from changing too drastically in a single step.

However, these local approximations are prone to surrogate objective misalignment. In A Closer Look at Deep Policy Gradients, Ilyas demonstrated that these surrogate landscapes frequently diverge from the true reward landscape, particularly in low-sample regimes or during late-stage training.

Value / Reward
  ▲
  │         True Reward Landscape
  │             _..-─-.._
  │           _/. . . . .\_      Optimizing the surrogate objective
  │          / . . . . . . \     beyond its local validity region
  │         │. . . . . . . .│    leads to a drop in true performance.
  │         │. . . ┌───┐ . .│
  │        / . . . │Sur│. . .\
  │       │. . . . │rog│. . . │
  │_______│________│ate│_______│________________> Policy Space

When this misalignment occurs, maximizing the surrogate objective causes a literal decrease in the true reward. The optimization process exploits flaws in the local proxy, favoring policy changes that degrade actual performance.

Conclusion

The shift toward decision-aware, control-oriented reinforcement learning represents a core realization: accuracy in isolation is an illusion. Whether training a world model, a value critic, or a preference model, the loss functions must be mathematically tied to the ultimate control objective. By replacing global statistical proxies with unified, task-weighted, and gradient-aware objectives, modern RL frameworks ensure that every bit of computational capacity is spent on learning what truly matters for decision-making.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_objective_mismatch_c508a0e7,
  title = {objective mismatch},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/objective-mismatch/rdr_c508a0e7e9d54a8caa0cb3b3a5b2520e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_objective_mismatch_c508a0e7_v1,
  title = {objective mismatch},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/objective-mismatch/rdr_c508a0e7e9d54a8caa0cb3b3a5b2520e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake