IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning

Zefeng Liang, Jie Qiao, Ruichu Cai, Weilin Chen, Zhifeng HaoarXiv 2026 · arXiv

TL;DR

Model-based reinforcement learning (MBRL) often stumbles because an agent's own policy introduces confounding bias, leading the dynamics model to confuse action effects with natural environmental evolution. This paper introduces IADD-TR, a framework that cleanly decouples action interventions from natural physics using a "zero-action anchor" and applies targeted regularization to the critic to ensure unbiased, robust policy gradient updates.

What Problem Does This Paper Address?

In Model-Based Reinforcement Learning (MBRL), agents learn a transition model of the environment to generate synthetic data, drastically reducing the number of real-world interactions needed. However, the data used to train these models is collected by the agent's own evolving policy, which introduces severe confounding bias.

Consider a self-driving car approaching a destination: the policy typically accelerates early in the trajectory and brakes near the end. A standard, monolithic transition model trained on this data might mistakenly learn a spurious correlation—for instance, associating a specific position on the track with an inherent, natural deceleration, rather than attributing it to the action of braking.

This confounding bias corrupts MBRL in two ways:

  1. Dynamics Entanglement: The transition model fails to generalize because it mixes up the immediate impact of an action with the environment's natural physics.
  2. Critic Selection Bias: The value function (critic) is only trained on actions the policy frequently selects, making it highly inaccurate when evaluating "counterfactual" actions (what would happen if we took a rare action). This leads to poor policy updates.
Fig. 1. An example of confounding bias in a car-driving scenario, together with an illustration of intervention-aware dynamics decoupling and counter- factual reasoning for the Q-function.

Methods

To solve these twin issues, the authors propose a two-pronged framework: Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR).

1. Intervention-Aware Dynamics Decoupling (IADD)

Instead of treating the environment transition as a single black-box mapping, IADD splits it into two distinct stages:

  • Stage 1 (Action Intervention): Models the immediate, localized impact of the action, outputting an intermediate representation.
  • Stage 2 (Natural Evolution): Models how the environment evolves from that intermediate state purely based on its underlying physics, completely independent of the action.

To make this latent intermediate state mathematically identifiable and grounded in physical reality, the authors introduce a zero-action anchor ($a_0$). When the agent takes a "zero action" (e.g., coasting without pressing the gas or brake), the framework enforces that the intermediate state must exactly equal the current state. This simple constraint allows the model to isolate natural physics from action-driven changes.

2. Targeted Regularization (TR)

Standard critics minimize temporal-difference (TD) Bellman error, which treats all data equally. However, the actor only relies on the critic to get an accurate gradient direction for policy improvement.

TR re-aligns the critic's objective with the actor's goal. Derived from the Efficient Influence Function (EIF) in causal inference, TR augments the critic with an action-density-scaled residual correction. This guarantees double robustness: the policy gradient estimate remains consistent if either the critic value function or the estimated replay action density is correctly specified.

Fig. 2. Illustration of the proposed IADD-TR method. The framework consists of two key components: IADD and targeted regularization (TR). (a) In the overall training pipeline, the agent interacts with the environment to collect real transitions, which are used to train the dynamics model and to update the policy. The learned model further generates synthetic transitions for model-based policy optimization. (b) IADD adopts a two-stage dynamics model, which decomposes the transition prediction process into an intermediate-stage prediction and a final reward/next-state prediction. (c) TR modifies the critic learning objective by introducing an adjusted critic and a targeted regularization term, improving the robustness of value estimation under imperfect model rollouts.

Datasets

The authors evaluated their framework on two main setups:

  1. MuJoCo Continuous-Control Benchmarks: Five standard physics simulation tasks representing varying dimensional complexities: HalfCheetah, Hopper, Walker2d, Ant, and Humanoid.
  2. Controlled Synthetic Dynamics Environment: A custom-built, two-dimensional damped-oscillator toy system where the true post-intervention intermediate states are explicitly recorded. This was used strictly to verify whether the zero-action anchor successfully recovers true latent physical properties.

Key Insights and Ideas

  • Actions as Causal Interventions: Shifting the perspective of reinforcement learning transitions from basic function approximation to a causal intervention framework fundamentally changes how we should structure neural network architectures for dynamics.
  • The Power of the Anchor: Grounding latent variables using a physically meaningful "zero action" provides a non-parametric anchor that achieves pointwise identifiability without needing complex auxiliary loss functions or hyperparameter tuning.
  • Gradient-Centric Critic Tuning: Critics don't need to be perfect everywhere; they just need to be unbiased in the directions the actor wants to move. Optimizing the critic via the EIF score equation directly neutralizes the selection bias caused by uneven data coverage.

Key Results

  • Superior Sample Efficiency: IADD-TR consistently matched or outperformed state-of-the-art baselines like MBPO, SAC, and PPO across all MuJoCo tasks. The performance gains were most pronounced in high-dimensional environments like Ant and Humanoid, where compounding model errors usually degrade standard MBRL performance.
Ablation performance curves across the five MuJoCo tasks, demonstrating the complementary benefits of combining IADD and TR.
  • Decoupling Verification: In the synthetic environment, introducing a small fraction of zero-action data ($\lambda_0 = 0.10$) caused the post-intervention state error to plummet, eventually achieving a 99.7% reduction in Mean Squared Error (MSE) at higher ratios, proving that the model successfully isolated the true causal mechanics.
  • Directional Gradient Accuracy: Using cosine similarity metrics against a ground-truth Monte Carlo baseline, the authors showed that Targeted Regularization consistently provides a more directionally accurate policy gradient estimate, especially during the early and intermediate stages of training.

Limitations and Future Work

The primary limitation of IADD-TR is its reliance on the availability of a meaningful zero action ($a_0$). While intuitive in robotics and locomotion (e.g., applying zero torque), defining a "zero action" might be ambiguous or non-trivial in more abstract decision-making domains, such as financial trading or game-playing environments with discrete, non-physical action spaces. Future work could explore extending this causal anchoring concept to environments where a natural baseline action is not readily apparent.

Summary updated

Abstract

Model-based reinforcement learning (MBRL), which learns environment dynamics to generate synthetic experience, is a promising approach to sample-efficient decision making. Numerous methods have been developed to improve dynamics prediction and policy optimization for MBRL through uncertainty estimation, model regularization, and conservative value learning. However, these methods typically treat the transition model and critic as monolithic predictors, overlooking the policy-induced data bias. Consequently, action can become entangled with environmental evolution, while uneven action coverage may distort the counterfactual value estimates used for policy improvement. To address this, we propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR). IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization. Its latent and state-aligned components are identifiable up to an invertible within-block transformation and pointwise, respectively. For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional. TR augments the critic with an action-density-scaled residual correction and optimizes a targeted loss, yielding doubly robust policy-gradient estimation when either the critic or the replay action density is consistently specified. Extensive experiments on five MuJoCo tasks show that IADD-TR achieves competitive returns with improved sample efficiency.
BibTeX
@misc{liang2026iadd,
  author = {Zefeng Liang and Jie Qiao and Ruichu Cai and Weilin Chen and Zhifeng Hao},
  title = {{IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning}},
  year = {2026},
  eprint = {2608.10634},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.10634},
}
Figure