Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUp

Junfan Lin, Zhongzhan Huang, Keze Wang, Xiaodan Liang, Weiwei Chen, Liang Lin

Abstract

Although deep reinforcement learning (RL) has been successfully applied to a variety of robotic control tasks, it's still challenging to apply it to real-world tasks, due to the poor sample efficiency. Attempting to overcome this shortcoming, several works focus on reusing the collected trajectory data during the training by decomposing them into a set of policy-irrelevant discrete transitions. However, their improvements are somewhat marginal since i) the amount of the transitions is usually small, and ii) the value assignment only happens in the joint states. To address these issues, this paper introduces a concise yet powerful method to construct Continuous Transition, which exploits the trajectory information by exploiting the potential transitions along the trajectory. Specifically, we propose to synthesize new transitions for training by linearly interpolating the consecutive transitions. To keep the constructed transitions authentic, we also develop a discriminator to guide the construction process automatically. Extensive experiments demonstrate that our proposed method achieves a significant improvement in sample efficiency on various complex continuous robotic control problems in MuJoCo and outperforms the advanced model-based / model-free RL methods.

Introduction

Deep Reinforcement Learning (RL) has shown remarkable potential in mastering complex robotic control tasks. However, its practical application is often hindered by extreme "data hunger." In real-world robotics, collecting millions of interaction samples is not only time-consuming but can also lead to significant hardware wear and tear.

Standard off-policy algorithms, such as Soft Actor-Critic (SAC) and Twin Delayed DDPG (TD3), attempt to mitigate this by breaking down continuous trajectories into discrete transitions $(s_t, a_t, r_t, s_{t+1})$. While this allows for data reuse via a replay buffer, it treats the environment as a series of isolated snapshots. This discretization ignores the underlying continuous nature of physical systems, leaving vast "gaps" in the state-action space that the agent never explores during training.

Conceptual comparison of Trajectory, Discrete Transition, and the proposed Continuous Transition approach.

Bridging the Gaps with Continuous Transitions

To address the sparsity of discrete transitions, we propose Continuous Transition. This method synthesizes new training samples by linearly interpolating between consecutive transitions in a trajectory, effectively filling the gaps in the replay buffer.

MixUp for Reinforcement Learning

Inspired by the MixUp technique used in computer vision, we create synthetic "intermediate" transitions by mixing the states, actions, and rewards of two adjacent time steps ($T_t$ and $T_{t+1}$) using a ratio $\epsilon$. Unlike standard MixUp, which often pairs random data points, we focus exclusively on consecutive transitions. In physical systems, adjacent states are highly correlated; interpolating between them is far more likely to yield a physically plausible transition than mixing unrelated states.

The Manifold Constraint and Discriminator

A major risk in RL data augmentation is "hallucination"—creating transitions that violate the laws of physics. If an agent learns from "impossible" data, its value function becomes unreliable. We visualize this as the "manifold" problem: synthetic transitions must stay within the region of physically possible states (the green area in Figure 2).

Illustration of the manifold constraint. Synthetic transitions must remain within the physically plausible region (green) to be useful for training.

To ensure authenticity, we introduce an energy-based discriminator. This component learns the environment's dynamics and calculates a prediction error for every transition. We use this feedback to automatically tune the interpolation process. If the discriminator identifies a synthetic transition as "too fake" (high error), the system reduces the interpolation intensity, keeping the data points closer to the original, verified discrete transitions.

Experimental Evaluation

We evaluated our method on the MuJoCo physics engine across several benchmark tasks, including Ant-v2, HalfCheetah-v2, Walker2d-v2, and Hopper-v2.

The results demonstrate a substantial leap in sample efficiency. For instance, in the HalfCheetah task, integrating Continuous Transitions allowed SAC to reach an average return of approximately 8200 at 200k timesteps, a significant improvement over the baseline SAC's 5500. Furthermore, our method outperformed more complex state-of-the-art baselines like SUNRISE and PETS. Because Continuous Transition operates at the data level, it acts as a "plug-and-play" wrapper that can be easily integrated into existing off-policy RL frameworks without modifying the underlying algorithms.

Conclusion

Continuous Transition offers a simple yet effective way to exploit the inherent continuity of physical trajectories. By generating dense, physically plausible synthetic data, we enable RL agents to learn smoother value functions and generalize better to unseen states. This approach significantly reduces the sample requirements for complex control tasks, bringing deep RL one step closer to practical real-world robotics.

Summary updated

Abstract

Although deep reinforcement learning (RL) has been successfully applied to a variety of robotic control tasks, it's still challenging to apply it to real-world tasks, due to the poor sample efficiency. Attempting to overcome this shortcoming, several works focus on reusing the collected trajectory data during the training by decomposing them into a set of policy-irrelevant discrete transitions. However, their improvements are somewhat marginal since i) the amount of the transitions is usually small, and ii) the value assignment only happens in the joint states. To address these issues, this paper introduces a concise yet powerful method to construct Continuous Transition, which exploits the trajectory information by exploiting the potential transitions along the trajectory. Specifically, we propose to synthesize new transitions for training by linearly interpolating the consecutive transitions. To keep the constructed transitions authentic, we also develop a discriminator to guide the construction process automatically. Extensive experiments demonstrate that our proposed method achieves a significant improvement in sample efficiency on various complex continuous robotic control problems in MuJoCo and outperforms the advanced model-based / model-free RL methods. The source code is available.
BibTeX
@inproceedings{lin2021continuous,
  author = {Junfan Lin and Zhongzhan Huang and Keze Wang and Xiaodan Liang and Weiwei Chen and Liang Lin},
  title = {{Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUp}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9561137},
}
Figure