Introduction
Reinforcement Learning (RL) agents are notoriously brittle when faced with domain shift. An agent trained to perform a task in a specific visual environment often fails when minor changes, such as lighting or object color, are introduced. This problem is compounded by the difficulty of defining reward functions for complex tasks, leading researchers toward Imitation Learning (IL), where agents learn directly from expert demonstrations.
However, traditional IL methods like Generative Adversarial Imitation Learning (GAIL) suffer from training instability due to their adversarial nature. Furthermore, they rarely support zero-shot transfer—the ability to perform a task in a new "target" domain without additional training. This paper introduces a pipeline that combines disentangled feature learning with stable inverse reinforcement learning to address these challenges.
Methodology
The proposed framework merges two powerful concepts: DARLA (for robust representations) and IQ-Learn (for efficient imitation). The pipeline consists of four distinct stages:
- Expert Training: An expert policy is first trained using Proximal Policy Optimization (PPO) in a controlled source environment to generate high-quality trajectories.
- Disentangled Representation Learning (AnnealedVAE): To ignore domain-specific visual "noise," the agent learns a compressed latent representation of the world using an AnnealedVAE (a variation of the $\beta$-VAE). By gradually increasing the latent bottleneck's capacity, the model is forced to capture independent factors of variation, such as object position versus background color.
- Latent Space Imitation (IQ-Learn): The agent learns to mimic the expert within this abstract latent space using Inverse soft-Q Learning (IQ-Learn). Unlike adversarial methods, IQ-Learn learns a single Q-function that implicitly represents both the reward and the policy, avoiding min-max optimization instability.
- Zero-Shot Transfer: The final policy, which operates solely on latent factors, is deployed in a target environment. Because the VAE maps different visual styles to the same underlying latent factors, the agent can function without retraining.
Experimental Setup
The researchers evaluated the pipeline across three environments with varying degrees of domain shift:
- Cartpole Combi: A classic task where cart and pole colors are changed between source and target domains.
- Cartpole Background: A more difficult variant where the background color in the target domain was never seen during training.
- Super Mario Bros: A complex transfer task where the agent is trained on levels 8-1 through 8-3 and tested on the unseen level 2-1.
Key Results
The performance of the proposed agent varied significantly based on the nature of the domain shift. In the Cartpole Combi environment, the agent achieved perfect transfer with a maximum reward of 500.0, significantly outperforming a standard PPO transfer (55.10). In the Cartpole Background task, the agent maintained strong performance (325.50), though it did not reach the expert's ceiling.
However, the method struggled with the Super Mario Bros environment, scoring 306.48—lower than even a random agent (353.46). This suggests that while disentangled representations can bridge visual "style" gaps (like color), they struggle when the "content" or geometry of the state space changes significantly, as seen in new game levels.
Discussion and Limitations
The core insight of this work is that separating "physics" (what is happening) from "style" (how it looks) is a viable path for zero-shot transfer. However, the failure in Super Mario highlights the "Disentanglement Trap": VAEs do not always isolate factors perfectly and may over-index on background features. Furthermore, unsupervised disentanglement remains theoretically challenging without specific inductive biases. Future research may explore multi-shot transfer or training on a broader variety of environments to help the VAE identify truly invariant features across more complex state distributions.
Summary updated