Training robots directly in the real world is notoriously slow, expensive, and potentially unsafe. While simulators offer a practical alternative, transferring learned policies from simulation to reality is hindered by a massive "reality gap." Most current research focuses on visual sim-to-real, where a policy is trained on rendered images in simulation and deployed on real-world camera feeds. However, high-quality image rendering in simulators is computationally expensive—often 20x slower than state-based simulation—and building realistic rendering engines is a laborious task.
To address this, researchers propose a more cost-efficient approach: train a policy on cheap, low-dimensional states (like joint angles and velocities) and deploy it directly on high-dimensional images (camera pixels). This presents a challenging cross-modal domain adaptation problem. To solve it, the authors introduce CODAS (Cross-mOdal Domain Adaptation with Sequential structure), a framework that leverages the sequential nature of reinforcement learning (RL) trajectories to make sim-to-real transfer significantly cheaper and faster.
The Core Problem: Ill-Posed Mapping
Previous methods that attempt to match the distribution of states to images often fail because the mapping is inherently "ill-posed." If you only look at a single snapshot in time, multiple different states might look "reasonable" for a given image.
As illustrated above, an image $o_t$ could be mapped to a correct state $s_t$ or an incorrect state $s'_t$ that has a similar probability under a standard distribution matching objective. When the mapping crosses modalities (from images to states), it loses the local feature consistency (like colors and textures) that same-modal adaptation relies on, making the alignment task much harder. This ambiguity can lead the agent to take incorrect or unsafe actions.
The CODAS Framework
To overcome this ill-posed mapping, CODAS introduces several key innovations centered around the temporal dynamics of RL tasks:
- Sequential Trajectory Matching: Instead of treating every image-state pair as an independent data point, CODAS treats them as sequences. By looking at the "context" of a trajectory, the model can rule out incorrect state mappings that might look plausible in a single frame but violate the laws of physics over time. Sequential structure acts as the ultimate tie-breaker.
- Variational Inference Framework: The problem is formulated as a variational inference task. The goal is to learn an inference function (the mapping from images to states) that approximates the ground-truth posterior, optimized using a decomposed Evidence Lower Bound (ELBO).
- Adversarial Training: To align the distributions, CODAS employs a GAN-style objective. A discriminator attempts to distinguish between "real" trajectories from the source domain (states) and "mapped" trajectories from the target domain (images).
- Embedded Dynamics Model (DM): To stabilize training over long horizons, the authors incorporate a residual structure. A pre-trained dynamics model provides an "average" estimate of the next state, and the neural network only has to learn the "correction" (residual). This approach is inspired by the success of ResNets in deep learning.
Experiments and Key Results
The method was rigorously evaluated on two major benchmarks: six classic control environments from MuJoCo (e.g., Hopper, HalfCheetah, Walker2d) and four complex tasks from the Robot Hand Manipulation Suite (Adroit) (e.g., opening a door, spinning a pen). In all experiments, the source domain consisted of low-dimensional states, and the target domain consisted of rendered images.
- Superior Performance: CODAS consistently outperformed standard GANs, CycleGANs, and Behavior Cloning baselines. On MuJoCo tasks, the adapted policies achieved roughly 85% of the performance of an "oracle" policy trained directly on states.
- State Mapping Accuracy: Quantitative results demonstrated that CODAS achieved a significantly lower Root Mean Squared Error (RMSE) when mapping images back to their ground-truth states compared to baseline methods.
- Robustness to Mismatch: CODAS exhibited a surprising tolerance to physical mismatches (e.g., different friction coefficients between the simulator and the real world) and differences in the policies used to collect data, largely due to its robust distributional modeling approach.
Limitations and Future Work
While highly effective, CODAS does have limitations. It struggled with the "Relocate" task in the Hand Manipulation Suite, likely due to the task's goal-conditional nature (the ball starts in a random spot), which creates a highly complex data distribution requiring more data to map accurately. Additionally, while robust to minor physics variations, the method's performance degrades if the reality gap exceeds a certain threshold.
Future work may involve extending CODAS to explicitly handle dynamics model mismatches within the framework, and applying these sequential constraints to standard image-to-image sim-to-real tasks to further improve their stability and reliability.
Summary updated