In traditional reinforcement learning (RL) and imitation learning (IL), algorithms often rely on joint, simultaneous optimization loops. In actor-critic frameworks, the actor updates its policy while the critic simultaneously updates its value function. In adversarial imitation learning, a policy updates to fool a discriminator that is itself constantly adapting. While elegant in theory, this co-training creates a notorious moving target problem. As both components shift concurrently, training becomes highly non-stationary, leading to optimization oscillations, mode collapse, and severe sample inefficiency.
To solve this, a powerful paradigm has emerged: decoupling. By separating value estimation, reward learning, or constraint tracking from the policy optimization loop into distinct, non-interleaved stages, researchers transform unstable bi-level or minimax optimization problems into stable, sequential regression and policy extraction steps.
Breaking the Adversarial Loop in Imitation Learning
Imitation learning traditionally uses adversarial frameworks to match an expert's state-action distribution. However, joint adversarial training is notoriously finicky. Decoupled imitation learning eliminates this instability by splitting the process into two independent phases: first, estimating a static reward function from expert data, and second, optimizing a policy against this fixed reward using standard RL.
For instance, Ruohan Wang et al. introduced this concept in Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation, which pre-computes a fixed reward function based on expert policy support estimation before any policy optimization occurs. Similarly, Minghuan Liu proposed Energy-Based Imitation Learning, which learns a stationary energy-based representation of expert behavior first, completely bypassing the iterative "cat-and-mouse" game of adversarial setups.
This non-iterative approach has been extended across various modalities:
- Optimal Transport: Yicheng Luo et al. developed Optimal Transport for Offline Imitation Learning to pre-label offline datasets with static rewards by aligning them to expert demonstrations, allowing practitioners to plug in any robust offline RL backbone.
- Successor Representations: Seyed Mahdi B. Azad et al. decoupled reward estimation from policy optimization in SR-Reward: Taking The Path More Traveled by leveraging Successor Representations to estimate state-action visitations conservatively.
- Denoising Landscapes: Anish Abhijit Diwan utilized generative energy-based models in Noise-conditioned Energy-based Annealed Rewards (NEAR): A Generative Framework for Imitation Learning from Observation to pre-train stationary reward functions that gradually denoise as the agent improves.
Even when starting from a supervised baseline, decoupling provides a cleaner path. Joe Watson et al. showed in Coherent Soft Imitation Learning that a shaped reward can be directly extracted from a behavioral-cloned policy, allowing subsequent RL fine-tuning without the risk of unlearning expert behavior or dealing with complex saddle-point optimization.
This philosophy also stabilizes sequence generation and dialogue systems. The authors of Expert-based reward function training: the novel method to train sequence generators replaced volatile GAN frameworks with a reward function trained on a fixed proposal distribution. Meanwhile, Ziming Li et al. in Guided Dialogue Policy Learning without Adversarial Learning in the Loop and Bang Liu et al. in Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management demonstrated that pre-training dialogue reward models offline prevents the mode collapse typical of in-the-loop adversarial training.
Eliminating Critic Drift in Actor-Critic Frameworks
The moving target problem is equally severe in actor-critic optimization, where the critic's value estimates drift as the actor's policy changes. Decoupled value-policy optimization breaks this dependency by ensuring the value function or critic is either optimized to convergence before the policy updates, or trained to target the optimal value function ($Q^*$) rather than the current policy's value ($Q^\pi$).
In offline goal-conditioned reinforcement learning (GCRL), Yecheng Jason Ma et al. demonstrated in Offline Goal-Conditioned Reinforcement Learning via $f$-Advantage Regression that optimizing the value function to convergence before beginning policy training provides a highly stable learning signal. This insight forms the basis of the two-step framework by Hanlin Zhu et al. in Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability, which optimizes the value function first and subsequently extracts the policy via weighted maximum likelihood estimation. Similarly, Yecheng Jason Ma et al. leveraged Fenchel duality in Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching to transform constrained state-matching into a stable, unconstrained value estimation phase that is entirely decoupled from policy updates.
For large language models (LLMs) and complex reasoning tasks, simultaneous co-training is computationally prohibitive and unstable. Chenghua Huang et al. introduced Pretrain Value, Not Reward: Decoupled Value Policy Optimization, which replaces the traditional online critic with a frozen, offline-pretrained Global Value Model to provide stable advantage estimates. In multi-step reasoning, Jiacai Liu et al. showed in DAPO : Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization that freezing a standalone critic trained on pre-computed value estimates prevents the moving-target problem and significantly reduces memory overhead during the actor's update phase.
Even in online settings requiring strategic exploration, targeting the optimal value function directly provides a more robust learning signal:
- Diederik M Roijers et al. developed Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics, forcing critics to learn $Q^*$ directly rather than evaluating the current suboptimal actor to accelerate convergence.
- Kevin Tan et al. proved in Actor-Critics Can Achieve Optimal Sample Efficiency that an actor-critic algorithm can achieve optimal sample efficiency by targeting $Q^*$ and employing rare-switching updates, mathematically bridging the gap between practical actor-critic popularity and rigorous guarantees.
This echoes early foundational work by Andras Antos et al. in Fitted Q-iteration in continuous action-space MDPs, where learning alternated between discrete policy searches and least-squares value updates using a decoupled target generation process.
Extending Decoupling to Constraints, Safety, and Data Backbones
The core intuition of decoupling—isolating interacting components to prevent error propagation—extends naturally to safety constraints, data preprocessing, and model architectures.
In constrained MDPs, estimating rewards and safety costs simultaneously can cause estimation errors in one signal to corrupt the other. To prevent this, Sharan Vaswani et al. evaluated the objective function and constraint costs using entirely separate subroutines in Sample Complexity Bounds for Linear Constrained MDPs with a Generative Model. For offline safety-critical tasks like medical interventions, Dipesh Tamboli et al. trained a high-accuracy mortality classifier independently and used it as a frozen critic in Reinforced Sequential Decision-Making for Sepsis Treatment: The POSNEGDM Framework with Mortality Classifier and Transformer, ensuring the decision-making agent received a stable risk signal.
Decoupling also resolves architectural and data-level tensions across several specialized domains:
- Training vs. Evaluation Constraints: Linjie Xu et al. in Mildly Constrained Evaluation Policy for Offline Reinforcement Learning decoupled the policy used for value updates from the policy used for final execution, applying strict constraints during training for stability but relaxed constraints during evaluation to maximize performance.
- Data Purification: When dealing with noisy demonstrations, Yunke Wang et al. treated data cleaning as a separate preprocessing step via diffusion models in Imitation Learning from Purified Demonstrations, avoiding the complex bi-level optimization common in weighted imitation learning. Similarly, Marzi Heidari et al. stabilized label correction in Learning to Clean: Reinforcement Learning for Noisy Label Correction by decoupling the policy network from the feature extraction backbone.
- Data Augmentation: To prevent high-variance augmented inputs from destabilizing value estimation, Abdulaziz Almuzairee et al. in A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learning anchored the target Q-function to unaugmented images while training the policy on strongly augmented data.
- Multi-Stage Planning: In generative planning, Xintong Duan et al. trained the reward model, teacher diffusion model, and student consistency model in completely separate stages in Accelerating Diffusion Planners in Offline RL via Reward-Aware Consistency Trajectory Distillation, eliminating the moving target issues of joint actor-critic training.
Ultimately, whether applied to rewards, values, or safety constraints, decoupling works because it replaces a chaotic, co-dependent dynamical system with a predictable sequence of isolated optimization problems. By giving each component a fixed target, the entire learning process becomes robust, stable, and theoretically sound.
Key Foundational Work
Fitted Q-iteration in continuous action-space MDPs (2007) — contributed 1 concept elements to this direction.
Safely Interruptible Agents (2016) — contributed 1 concept elements to this direction.