Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUp

Junfan Lin, Zhongzhan Huang, Keze Wang, Xiaodan Liang, Weiwei Chen, Liang Lin

TL;DR

This paper introduces Continuous Transition, a data augmentation technique for reinforcement learning that synthesizes new training samples by linearly interpolating consecutive transitions. By using an energy-based discriminator to keep these "MixUp" samples physically plausible, the authors significantly improve the sample efficiency of standard algorithms like SAC and TD3 on complex robotic control tasks.

What Problem Does This Paper Address?

Deep Reinforcement Learning (RL) is notoriously "data-hungry." In robotics, collecting millions of interactions is often expensive, slow, or physically damaging to the hardware.

Current off-policy methods (like SAC or TD3) try to improve efficiency by breaking down continuous trajectories into a set of discrete, policy-irrelevant transitions $(s_t, a_t, r_t, s_{t+1})$. However, this approach treats the environment as a series of "snapshots." It ignores the underlying continuous nature of physical movement, leaving "gaps" in the state-action space that the agent never sees during training. This leads to poor generalization: the agent's value function might behave erratically when it encounters a state slightly different from the ones in its replay buffer, a problem known as being "out-of-manifold."

Methods

The authors propose a method to fill these gaps by creating a "dense" stream of data from a "sparse" trajectory.

  1. Continuous Transition via MixUp: Inspired by the MixUp technique in computer vision, the authors linearly interpolate between two consecutive transitions ($T_t$ and $T_{t+1}$). By mixing the states, actions, and rewards of two adjacent time steps using a ratio $\epsilon$, they create a synthetic "intermediate" transition.
  2. Why Consecutive? Unlike standard MixUp, which often mixes random pairs of data, this paper argues that consecutive transitions in a physical system share similar dynamics. Interpolating between them is much more likely to produce a transition that could actually happen in the real world.
  3. Energy-Based Discriminator: To prevent the model from learning from "impossible" transitions (hallucinations), they introduce a discriminator. This is an MLP that learns the environment's dynamics. It calculates a "prediction error" for any given transition.
  4. Automatic Temperature Tuning: The interpolation ratio $\epsilon$ is sampled from a Beta distribution $B(\beta, \beta)$. The authors don't just pick a fixed $\beta$; they use the discriminator's feedback to auto-tune it. If the synthetic transitions look "too fake" (high prediction error), the system lowers the "temperature" to stay closer to the original discrete data points.

Datasets

The method was evaluated on the MuJoCo (Multi-Joint dynamics with Contact) physics engine, a standard benchmark for continuous control in RL. Specifically, they tested on:

  • Ant-v2
  • HalfCheetah-v2
  • Walker2d-v2
  • Hopper-v2

These environments represent various levels of complexity in high-dimensional state and action spaces.

Key Insights and Ideas

  • Density over Quantity: The core "aha moment" is that RL agents don't just need more data; they need denser data. By interpolating, the agent learns a smoother value function, which prevents it from over-memorizing specific state-action pairs.
  • The Manifold Constraint: Data augmentation in RL is dangerous because the "labels" (next states and rewards) are governed by physics, not just human logic. The use of a discriminator to keep synthetic data "in-manifold" is a clever way to provide the benefits of model-based RL (synthetic data) with the simplicity of model-free RL.
  • Plug-and-Play: Because this method operates at the data level (the replay buffer), it requires no changes to the underlying RL algorithm. You can simply "wrap" SAC or TD3 with Continuous Transitions.

Key Results

The results show a massive jump in sample efficiency across all MuJoCo tasks:

  • Performance Boost: On the "Cheetah" task, SAC with Continuous Transitions (SAC+c.t.) reached an average return of ~8200, compared to the original SAC's ~5500 at 200k timesteps.
  • Beating Ensembles: The method outperformed complex state-of-the-art baselines like SUNRISE and PETS, which use ensembles of models or complex planning, despite being much more computationally efficient.
  • Early Learning: The evaluation curves show that the improvement happens very early in the training process, suggesting that the "denser" data is particularly helpful when the replay buffer is still relatively empty.

Limitations and Future Work

  • Sim-to-Real: While MuJoCo is a great benchmark, the paper does not test this on physical robots where sensor noise and non-linearities are more prevalent.
  • State Representation: The current method works on feature-based inputs (velocity, position). The authors note that applying this to pixel-based (vision) RL is a logical next step but might require different interpolation strategies.
  • Interpolation Limits: The paper focuses on consecutive transitions. Future work could explore if interpolating between "nearby" states (found via nearest-neighbor search in the replay buffer) could further increase data variety without breaking physical plausibility.

Summary updated

Abstract

Although deep reinforcement learning (RL) has been successfully applied to a variety of robotic control tasks, it’s still challenging to apply it to real-world tasks, due to the poor sample efficiency. Attempting to overcome this shortcoming, several works focus on reusing the collected trajectory data during the training by decomposing them into a set of policy-irrelevant discrete transitions. However, their improvements are somewhat marginal since i) the amount of the transitions is usually small, and ii) the value assignment only happens in the joint states. To address these issues, this paper introduces a concise yet powerful method to construct Continuous Transition, which exploits the trajectory information by exploiting the potential transitions along the trajectory. Specifically, we propose to synthesize new transitions for training by linearly interpolating the consecutive transitions. To keep the constructed transitions authentic, we also develop a discriminator to guide the construction process automatically. Extensive experiments demonstrate that our proposed method achieves a significant improvement in sample efficiency on various complex continuous robotic control problems in MuJoCo and outperforms the advanced model-based / model-free RL methods. The source code is available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
BibTeX
@inproceedings{lin2021continuous,
  author = {Junfan Lin and Zhongzhan Huang and Keze Wang and Xiaodan Liang and Weiwei Chen and Liang Lin},
  title = {{Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUp}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9561137},
}

Loading related content...

Figure