Discovering Object-Centric Generalized Value Functions From Pixels

Somjit Nath, Gopeshh Raaj Subbaraj, Khimya Khetarpal, Samira Ebrahimi Kahou

TL;DR

This paper introduces Object-Centric Generalized Value Functions (OC-GVFs), a novel method for Reinforcement Learning (RL) agents to learn meaningful representations directly from pixel inputs. By combining object-centric feature extraction with the discovery of "question" functions (cumulants), OC-GVFs enable faster adaptation and improved performance in complex, non-stationary environments.

What Problem Does This Paper Address?

Deep Reinforcement Learning (RL) agents often struggle to learn effectively from high-dimensional inputs like images. While auxiliary tasks and pseudo-rewards have shown promise in improving representation learning, automatically discovering these tasks in an object-centric manner, crucial for control and adaptation, remains an open challenge. Existing methods often rely on hand-crafted features or struggle to generalize to new, unseen situations, especially when the environment's dynamics change (non-stationarity). This paper aims to bridge this gap by developing a method that learns to identify important objects and their properties from pixels, using this knowledge to inform the agent's decision-making process.

Methods

The core of the proposed method, OC-GVFs, lies in a two-part architecture:

  • Question Network: This network uses a slot attention mechanism to process input states (images) and extract object-centric representations. Each "slot" is designed to capture features of a distinct object in the scene. These slots are then used to predict cumulants, which are essentially scalar reward signals that define "questions" for the Generalized Value Functions (GVFs). The slot attention is trained using a reconstruction loss, encouraging it to learn meaningful object representations.
  • Main Network: This network is a standard RL agent (specifically, a Double DQN agent in their experiments) that learns the main task policy. Crucially, it uses the GVFs (the "answers" to the questions posed by the cumulants) as features. These GVFs are learned by the main network using the cumulants provided by the question network. The integration of GVFs as features, along with layer normalization and concatenation in the latent space, contributes to improved stability and faster learning.

The key novelty lies in the object-centric discovery of cumulants through slot attention, which are then used to learn GVFs that serve as rich, interpretable features for the main RL agent. This decouples object discovery from the main RL loss, allowing for more robust learning.

Datasets

The experiments were conducted on several simulated environments:

  • Collect-objects Environment: A customized gridworld environment where the agent must collect objects of specific colors in a particular order. A non-stationary version was also used where object locations changed randomly each episode.
  • MiniGrid-Dynamic Obstacles: A MiniGrid environment with randomly placed obstacles.
  • CoinRun & StarPilot: Procedurally generated environments from the ProcGen benchmark, used to test adaptation to new, unseen levels.

Key Insights and Ideas

  • Object-centricity is key for adaptation: By explicitly learning representations tied to objects, the agent can better adapt to changes in the environment, as object properties often remain consistent even when their positions or task dynamics change.
  • GVFs as features are powerful: Integrating learned GVFs as features into the main RL agent's representation significantly boosts performance, especially in complex or non-stationary settings. This suggests that the predictive knowledge captured by GVFs is highly valuable for decision-making.
  • Slot attention for unsupervised object discovery: The use of slot attention allows for the unsupervised discovery of object-centric features from raw pixels, eliminating the need for hand-crafted object detectors. This is a crucial step towards more generalizable RL agents.
  • Decoupling object discovery from RL loss: Separating the learning of object representations (via reconstruction loss) from the learning of GVFs (via RL loss) leads to more stable and effective cumulant discovery, especially for non-stationarity.

Key Results

  • Improved sample efficiency: OC-GVFs demonstrated significantly better sample efficiency compared to baselines like DDQN, Random-GVFs, and Dis-Aux-GVFs in both stationary and non-stationary settings.
  • Fast adaptation to non-stationarity: In non-stationary environments (e.g., changing object locations), OC-GVFs showed a much faster convergence rate than baselines, highlighting their ability to adapt to dynamic changes.
  • Effective transfer learning: The method showed promising results in adapting to new, unseen tasks (levels in CoinRun and StarPilot) with minimal prior information, outperforming baselines in scenarios with increasing complexity.
  • Qualitative insights: Visualizations showed that learned GVFs indeed captured object-specific properties, often reflecting a "distance metric" to objects, making the learned representations interpretable.

Limitations and Future Work

  • Reliance on slot attention: The method's performance is dependent on the slot attention mechanism's ability to effectively segment objects. If objects are difficult to distinguish (e.g., similar colors, constant movement), the method may struggle.
  • Color-based object separation: Slot attention's reliance on color for object separation can be problematic in scenarios where objects have similar colors.
  • Future Directions:
    • Exploring zero-shot transfer learning with previously learned cumulants, even with similar objects in new environments.
  • Investigating task-agnostic losses for cumulant training to potentially improve generalization.
  • Extending the framework to learn the discount factor ($\gamma$) and policy directly.

Summary updated

Abstract

Deep Reinforcement Learning has shown significant progress in extracting useful representations from high-dimensional inputs albeit using hand-crafted auxiliary tasks and pseudo rewards. Automatica...
BibTeX
@inproceedings{nath2023discovering,
  author = {Somjit Nath and Gopeshh Raaj Subbaraj and Khimya Khetarpal and Samira Ebrahimi Kahou},
  title = {{Discovering Object-Centric Generalized Value Functions From Pixels}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=8AQrtiuujE},
}
Figure