Cross-modal Domain Adaptation for Cost-Efficient Visual Reinforcement Learning

Xiong-Hui Chen, Shengyi Jiang, Feng Xu, Zongzhang Zhang, Yang YuNeurIPS 2021 · OpenReview

Training robots directly in the real world is notoriously slow, expensive, and potentially unsafe. While simulators offer a practical alternative, transferring learned policies from simulation to reality is hindered by a massive "reality gap." Most current research focuses on visual sim-to-real, where a policy is trained on rendered images in simulation and deployed on real-world camera feeds. However, high-quality image rendering in simulators is computationally expensive—often 20x slower than state-based simulation—and building realistic rendering engines is a laborious task.

To address this, researchers propose a more cost-efficient approach: train a policy on cheap, low-dimensional states (like joint angles and velocities) and deploy it directly on high-dimensional images (camera pixels). This presents a challenging cross-modal domain adaptation problem. To solve it, the authors introduce CODAS (Cross-mOdal Domain Adaptation with Sequential structure), a framework that leverages the sequential nature of reinforcement learning (RL) trajectories to make sim-to-real transfer significantly cheaper and faster.

The Core Problem: Ill-Posed Mapping

Previous methods that attempt to match the distribution of states to images often fail because the mapping is inherently "ill-posed." If you only look at a single snapshot in time, multiple different states might look "reasonable" for a given image.

Comparison of mapping with and without sequential structure.

As illustrated above, an image $o_t$ could be mapped to a correct state $s_t$ or an incorrect state $s'_t$ that has a similar probability under a standard distribution matching objective. When the mapping crosses modalities (from images to states), it loses the local feature consistency (like colors and textures) that same-modal adaptation relies on, making the alignment task much harder. This ambiguity can lead the agent to take incorrect or unsafe actions.

The CODAS Framework

To overcome this ill-posed mapping, CODAS introduces several key innovations centered around the temporal dynamics of RL tasks:

  • Sequential Trajectory Matching: Instead of treating every image-state pair as an independent data point, CODAS treats them as sequences. By looking at the "context" of a trajectory, the model can rule out incorrect state mappings that might look plausible in a single frame but violate the laws of physics over time. Sequential structure acts as the ultimate tie-breaker.
  • Variational Inference Framework: The problem is formulated as a variational inference task. The goal is to learn an inference function (the mapping from images to states) that approximates the ground-truth posterior, optimized using a decomposed Evidence Lower Bound (ELBO).
Graphical model illustrating the sequential dependencies between states, observations, and actions.
  • Adversarial Training: To align the distributions, CODAS employs a GAN-style objective. A discriminator attempts to distinguish between "real" trajectories from the source domain (states) and "mapped" trajectories from the target domain (images).
  • Embedded Dynamics Model (DM): To stabilize training over long horizons, the authors incorporate a residual structure. A pre-trained dynamics model provides an "average" estimate of the next state, and the neural network only has to learn the "correction" (residual). This approach is inspired by the success of ResNets in deep learning.

Experiments and Key Results

The method was rigorously evaluated on two major benchmarks: six classic control environments from MuJoCo (e.g., Hopper, HalfCheetah, Walker2d) and four complex tasks from the Robot Hand Manipulation Suite (Adroit) (e.g., opening a door, spinning a pen). In all experiments, the source domain consisted of low-dimensional states, and the target domain consisted of rendered images.

  • Superior Performance: CODAS consistently outperformed standard GANs, CycleGANs, and Behavior Cloning baselines. On MuJoCo tasks, the adapted policies achieved roughly 85% of the performance of an "oracle" policy trained directly on states.
  • State Mapping Accuracy: Quantitative results demonstrated that CODAS achieved a significantly lower Root Mean Squared Error (RMSE) when mapping images back to their ground-truth states compared to baseline methods.
  • Robustness to Mismatch: CODAS exhibited a surprising tolerance to physical mismatches (e.g., different friction coefficients between the simulator and the real world) and differences in the policies used to collect data, largely due to its robust distributional modeling approach.

Limitations and Future Work

While highly effective, CODAS does have limitations. It struggled with the "Relocate" task in the Hand Manipulation Suite, likely due to the task's goal-conditional nature (the ball starts in a random spot), which creates a highly complex data distribution requiring more data to map accurately. Additionally, while robust to minor physics variations, the method's performance degrades if the reality gap exceeds a certain threshold.

Future work may involve extending CODAS to explicitly handle dynamics model mismatches within the framework, and applying these sequential constraints to standard image-to-image sim-to-real tasks to further improve their stability and reliability.

Summary updated

Abstract

In visual-input sim-to-real scenarios, to overcome the reality gap between images rendered in simulators and those from the real world, domain adaptation, i.e., learning an aligned representation space between simulators and the real world, then training and deploying policies in the aligned representation, is a promising direction. Previous methods focus on same-modal domain adaptation. However, those methods require building and running simulators that render high-quality images, which can be difficult and costly. In this paper, we consider a more cost-efficient setting of visual-input sim-to-real where only low-dimensional states are simulated. We first point out that the objective of learning mapping functions in previous methods that align the representation spaces is ill-posed, prone to yield an incorrect mapping. When the mapping crosses modalities, previous methods are easier to fail. Our algorithm, Cross-mOdal Domain Adaptation with Sequential structure (CODAS), mitigates the ill-posedness by utilizing the sequential nature of the data sampling process in RL tasks. Experiments on MuJoCo and Hand Manipulation Suite tasks show that the agents deployed with our method achieve similar performance as it has in the source domain, while those deployed with previous methods designed for same-modal domain adaptation suffer a larger performance gap.
BibTeX
@inproceedings{chen2021cross,
  author = {Xiong-Hui Chen and Shengyi Jiang and Feng Xu and Zongzhang Zhang and Yang Yu},
  title = {{Cross-modal Domain Adaptation for Cost-Efficient Visual Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=VLQV2vqjLf3},
}
Figure