Weakly Supervised Disentangled Representation for Goal-Conditioned Reinforcement Learning

Zhifeng Qian, Mingyu You, Hongjun Zhou, Bin He

TL;DR

This paper introduces DR-GRL, a framework that uses weakly supervised disentangled representation learning to significantly improve sample efficiency and policy generalization in goal-conditioned reinforcement learning. By learning to separate object attributes like shape, color, and position, their method can generate novel goals and more effectively learn policies for robotic manipulation tasks, even transferring them to real-world robots.

What Problem Does This Paper Address?

Goal-conditioned reinforcement learning (GRL) aims to train agents that can achieve a variety of user-specified goals. While powerful, current GRL methods often require an enormous number of environmental interactions (millions) to learn effectively, making them highly sample-inefficient, especially when dealing with complex visual inputs. A key challenge is generating diverse and meaningful goals for the agent to train on. Existing methods struggle with creating goals that are physically realistic, controllable, and generalize to unseen scenarios. This paper tackles the problem of sample inefficiency and poor generalization in visual GRL by proposing a novel approach to learn interpretable and controllable representations of objects.

Methods

The core of DR-GRL is a Spatial Transform Autoencoder (STAE), designed to learn disentangled representations of object attributes (shape, color, position) in a weakly supervised manner.

  • STAE Architecture: It takes an image as input and aims to disentangle its latent representation into distinct components for shape, color, and position. A key innovation is the Spatial Transform (ST) module, which explicitly learns translation parameters to manipulate object positions, ensuring the learned position representation is consistent with physical space. This contrasts with many prior works that learn abstract representations.
  • Weakly Supervised Progressive Swap Strategy: To train STAE without explicit attribute labels, the authors propose a three-stage strategy:
  • Zero Swap: Standard autoencoder reconstruction loss to ensure representations capture sufficient image information.
  • One Swap: Pairs of images with one shared attribute (e.g., same color) are used. Their attribute representations are swapped, and the network must reconstruct the original images. This forces the network to learn attribute-specific representations.
  • Cycle Swap: This stage involves a more complex swapping and reconstruction process, further encouraging disentanglement.
  • Goal Generation and Relabeling: The disentangled representations learned by STAE are crucial for GRL. The controllable position representation allows DR-GRL to generate novel goals by recombining shape/color representations and sampling new positions, even those outside the training distribution. These generated goals are then used to relabel data in the replay buffer, effectively creating more training samples without new environment interactions.
  • Goal-Conditioned RL Algorithm: The authors adapt Soft Actor-Critic (SAC), a state-of-the-art off-policy RL algorithm, to work with the learned representations. The learned position representation serves as an observation embedding, and the reward function is defined based on the Euclidean distance in the disentangled position representation space, which is shown to be consistent with physical distance.

Datasets

The experiments were conducted using a custom dataset of cups with varying shapes, colors, and positions, collected in a simulated environment (Mujoco's UR Pusher) and also in the real world using a UR5 robotic arm. The paper does not introduce a new benchmark dataset but rather uses a controlled environment for evaluation.

Key Insights and Ideas

  • Disentanglement for Controllability: The key insight is that learning disentangled representations, particularly those that explicitly model physical attributes like position, is crucial for generating controllable and diverse goals in GRL.
  • Spatial Transform Module's Importance: The ST module is vital for grounding the learned position representation in physical reality, enabling accurate reward calculation and effective goal generation.
  • Weak Supervision is Sufficient: The paper demonstrates that learning disentangled representations for complex tasks like robotic manipulation doesn't require explicit, per-instance attribute labels. A weakly supervised approach using reconstruction losses and attribute swapping is effective.
  • Relabeling with Imagined Goals: Generating novel goals from learned representations and using them to relabel past experiences is a powerful technique to boost sample efficiency, effectively allowing the agent to "imagine" more training data.
  • Simulation-to-Real Transfer: The disentangled representation, by focusing on object attributes and their physical relationships, facilitates seamless transfer of learned policies from simulation to the real world without extensive fine-tuning.

Key Results

  • Improved Sample Efficiency: DR-GRL significantly outperforms prior methods (RIG, CURL, CCRIG) in terms of sample efficiency, converging much faster in simulation.
  • Enhanced Policy Generalization: DR-GRL achieves higher success rates, especially on unseen cups (varying in shape and color), demonstrating superior generalization capabilities. For instance, it achieved an 85.33% success rate on seen cups and 84.00% on unseen cups in simulation, significantly outperforming others.
  • Consistent Physical Representation: Visualization using t-SNE shows that DR-GRL's learned representations maintain a clear manifold structure corresponding to physical positions, unlike other methods where representations are "messy."
  • Real-World Performance: The framework successfully transfers to a real UR5 robot, achieving a 73.33% success rate on seen cups and 70.00% on unseen cups, showcasing its practical applicability.
  • Ablation Studies: Experiments confirm the effectiveness of each component: the mixup operation helps handle robot pose variations, the ST module ensures position consistency, and uniform relabeling further boosts performance.

Limitations and Future Work

  • Complexity of Tasks: The authors acknowledge that DR-GRL is currently demonstrated on a relatively simple task (pushing cups). More complex tasks requiring attention to manipulator pose and simultaneous disentanglement of multiple objects remain challenging.
  • Multiple Objects: Extending the framework to handle scenes with multiple interacting objects is identified as an area for future research.
  • Unsupervised Detection: Investigating unsupervised object detection or exploiting sequential information could be beneficial for more complex scenarios.
  • Manipulator Pose: Future work could involve disentangling the manipulator's pose into the representation for more sophisticated control.

Summary updated

Abstract

Goal-conditioned reinforcement learning is a crucial yet challenging algorithm which enables agents to achieve multiple user-specified goals when learning a set of skills in a dynamic environment. However, it typically requires millions of the environmental interactions explored by agents, which is sample-inefficient. In the letter, we propose a skill learning framework DR-GRL that aims to improve the sample efficiency and policy generalization by combining the Disentangled Representation learning and Goal-conditioned visual Reinforcement Learning. In a weakly supervised manner, we propose a Spatial Transform AutoEncoder (STAE) to learn an interpretable and controllable representation in which different parts correspond to different object attributes (shape, color, position). Due to the high controllability of the representations, STAE can simply recombine and recode the representations to generate unseen goals for agents to practice themselves. The manifold structure of the learned representation maintains consistency with the physical position, which is beneficial for reward calculation. We empirically demonstrate that DR-GRL significantly outperforms the previous methods in sample efficiency and policy generalization. In addition, DR-GRL is also easy to expand to the real robot.
BibTeX
@article{qian2022weakly,
  author = {Zhifeng Qian and Mingyu You and Hongjun Zhou and Bin He},
  title = {{Weakly Supervised Disentangled Representation for Goal-Conditioned Reinforcement Learning}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  url = {https://openreview.net/forum?id=SP6XAzWfRl},
}
Figure