Abstract
Third-person imitation learning (TPIL) is a variant of generative adversarial imitation learning and can learn an expert-like policy from third-person expert demonstrations. Third-person expert demonstrations usually exist in the form of videos recorded in a third-person perspective, and there is a lack of direct correspondence with samples generated by the agent. To alleviate this problem, we improve TPIL by applying image difference and variational discriminator bottleneck. Empirically, our new method has better performance than TPIL on two MuJoCo tasks, Reacher and Inverted Pendulum.
Introduction
In standard Imitation Learning (IL), an agent typically learns from "first-person" demonstrations where the expert and agent share the same viewpoint. However, real-world applications often only provide "third-person" demonstrations, such as a robot learning from a human video. This introduces the correspondence problem: differences in camera angles, backgrounds, and lighting create a domain gap between the expert's observations and the agent's environment. Previous approaches to Third-Person Imitation Learning (TPIL) often required additional "non-expert" data from the expert's environment to bridge this gap, which is often impractical to collect. This work proposes a method to achieve high-performance TPIL without the need for such auxiliary datasets.
Methodology
The proposed approach, TPIL-ID-VDB, utilizes two key techniques to overcome the domain gap:
Image Difference (ID)
To mitigate the impact of static background elements, the system processes consecutive observations $(o_t, o_{t+n})$ by calculating their difference. This simple subtraction highlights motion regions relevant to the task while effectively "canceling out" domain-specific static features like background colors or fixed objects. This allows the agent to focus on motion as a universal language across different perspectives.
Variational Discriminator Bottleneck (VDB)
In adversarial imitation learning, a discriminator is trained to distinguish between expert and agent behaviors. However, the discriminator often "cheats" by identifying trivial perspective differences (e.g., a specific camera tilt) rather than actual behavioral differences. VDB addresses this by imposing an information constraint on the discriminator. By limiting the information flow, the discriminator is forced to discard noisy, domain-specific features and focus on essential, domain-agnostic behavioral characteristics.
Experimental Evaluation
The method was evaluated on two classic control tasks within the MuJoCo physics engine: Reacher (a robotic arm reaching a target) and Inverted Pendulum (balancing a pole). To simulate the third-person challenge, the researchers introduced variations in observing angles, background colors, and object colors between the expert demonstrations and the agent's training environment.
Key Results
As shown in the performance curves, TPIL-ID-VDB (black line) significantly outperforms the original TPIL (red line) and other variants. Notably:
- Efficiency: High performance is achieved without requiring additional non-expert demonstrations, a major improvement over previous TPIL frameworks.
- Performance Gap: While GAIL using first-person data (purple line) remains the upper bound, TPIL-ID-VDB closes the gap significantly compared to prior third-person methods.
- Robustness: The combination of ID and VDB allows the agent to ignore up to 90% of environmental "noise," focusing instead on high-level movement concepts that translate across viewpoints.
Conclusion and Future Work
This research demonstrates that combining motion-based feature extraction (ID) with information-constrained adversarial learning (VDB) provides a robust framework for third-person imitation learning. While successful in MuJoCo environments, future work will explore the scalability of this approach to more complex robotic manipulation tasks and highly dynamic backgrounds. Additionally, optimizing the adaptive update of the VDB's Lagrangian multiplier remains an area for further refinement.
Summary updated