TL;DR
Out-of-Dynamics Imitation Learning (OOD-IL) allows an agent to learn from demonstrators with different physical dynamics, but unachievable demonstrations and multimodal data distributions often derail learning. This paper introduces a two-phase framework that first clusters trajectories to untangle distinct behavioral modes and then uses adversarial learning to filter out non-transferable demonstrations. The result is a robust imitation learning pipeline that significantly outperforms state-of-the-art methods across robotic and driving simulations.
What Problem Does This Paper Address?
Standard imitation learning (IL) operates on a strict assumption: the student (imitator) and the teacher (demonstrator) must share the exact same dynamics, meaning they have identical bodies, action spaces, and environmental physics.
In the real world, this is rarely the case. We might want a 3-DoF robot arm to learn from a 7-DoF robot arm, or train an autonomous vehicle using human driving data collected from a completely different model of car in a different city. This setting is called Out-of-Dynamics Imitation Learning (OOD-IL).
The core challenge in OOD-IL is that some teacher behaviors are physically impossible for the student to replicate. Prior works attempted to filter out these "non-transferable" demonstrations using feasibility metrics. However, they overlooked a critical fact: datasets collected from multiple different demonstrators exhibit a multimodal distribution because different teachers use different strategies or operate under different constraints. Trying to fit a single, unimodal model to this messy mix of behaviors causes existing filtering techniques to fail.
Methods
To tackle the dual challenges of multimodality and non-transferability, the authors propose a two-phase framework:
- Sequence-based Contrastive Clustering: Before assessing feasibility, the framework untangles the multimodal dataset. It uses an LSTM-based feature extractor to map variable-length trajectories into a latent space. By treating sub-trajectories from the same demonstration as positive pairs and sub-trajectories from different demonstrations as negative pairs, the model learns a representation space where distinct behavioral modes naturally separate into $K$ clean clusters.
- Adversarial Transferability Measurement: Within each cluster, the algorithm isolates the behaviors and trains a Generative Adversarial Imitation Learning (GAIL) agent. The GAIL discriminator's job is to distinguish between the demonstrator’s transitions and the imitator’s current policy. Crucially, the discriminator's output serves as a proxy for transferability: if the discriminator believes a teacher's transition looks like something the student can successfully execute, that transition receives a high transferability score.
Finally, these scores are normalized into a sampling distribution, ensuring the final imitation learning policy trains predominantly on highly transferable data.
Datasets
The authors evaluated their approach across three distinct simulation setups:
- MuJoCo Benchmarks: Modified versions of HalfCheetah, Hopper, and Walker2d, where the source demonstrators had altered joint forces, gravity, or friction coefficients compared to the target imitator.
- Driving Environment: A custom 2D navigation task where a car must navigate around obstacles. The dataset included trajectories from cars with different speeds and obstacle configurations.
- Simulated Franka Panda Robot Arm: A 7-DoF robot arm environment implemented in PyBullet. The task was to push a box across a desk, with different source demonstrators having different individual joints disabled.
Key Insights and Ideas
- The Multimodal Trap: Previous state-of-the-art methods (like f-MDP) failed because they forced a single unimodal policy to fit a mixture of different demonstrator behaviors. This paper shows that clustering the data before filtering is essential.
- Discriminators as Filters: Instead of explicitly solving a complex and computationally slow inverse dynamics problem to check if an action is possible, the framework elegantly repurposes a standard GAIL discriminator to score transition feasibility.
- Robustness to Bad Data: The framework acts as an automated data-cleaning step, making it highly robust even when the dataset is heavily flooded with unachievable, non-transferable demonstrations.
Key Results
- Superior Performance: The proposed method consistently achieved the highest expected return across all MuJoCo, driving, and robotic arm environments, handily beating baselines like Naive GAIL and f-MDP.
- Compositional Stability: In stress tests where the ratio of non-transferable trajectories was drastically increased (e.g., from an equal 1:1 mix to a heavily skewed 1:50 mix), the final return of the proposed method remained stable, while baseline performances plummeted.
- Ablation Validation: Removing the clustering step (
Ours w/o Cluster) led to a significant drop in performance, experimentally validating that separating behavioral modes is key to accurate transferability estimation.
Limitations and Future Work
- Computational Scalability: Training a separate GAIL model for every single cluster can become computationally expensive if the dataset contains hundreds or thousands of distinct modes. The authors suggest pre-training or meta-learning as potential remedies.
- Safety Constraints: The framework does not explicitly handle safety boundaries during the exploration phases of transferability learning, which could be problematic for physical deployments.
- Sub-optimal Teachers: The method assumes that the collected demonstrations are optimal within their own source dynamics, leaving the challenge of learning from sub-optimal, multimodal OOD data to future work.
Summary updated