Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in First-person Simulated 3D Environments

Wilka Carvalho, Anthony Liang, Kimin Lee, Sungryull Sohn, Honglak Lee, Richard L. Lewis, Satinder Singh

TL;DR

This paper introduces LOAD, a reinforcement learning agent that learns complex object-interaction tasks in a simulated 3D environment using only sparse rewards. By training an "attentive object-model" as an auxiliary task, LOAD learns to represent objects and their relationships, significantly improving sample efficiency and performance compared to agents without this supervision.

What Problem Does This Paper Address?

Robotic agents need to perform intricate tasks involving multiple objects, like preparing food or tidying up. However, learning these tasks in realistic 3D environments is incredibly difficult when the agent only receives a reward signal upon successful task completion (sparse reward). This sparsity makes it hard for the agent to learn how to represent objects, their properties (e.g., "is it on?"), and how they relate to each other (e.g., "is something inside?"). Traditional approaches rely on expert demonstrations, shaped rewards, or ground-truth object information, which are often costly or impossible to obtain for new tasks. This paper tackles the challenge of learning these complex object-interaction tasks from scratch, without any of these forms of supervision.

Methods

The core of the proposed solution is the LOAD (Learning Object Attention & Dynamics) agent. LOAD combines two main components:

  • Attentive Object-DQN: This is an object-centric relational reinforcement learning policy. It uses an "inter-object attention" mechanism to consider how different objects in the scene relate to each other when deciding on an action. This is crucial because many tasks require understanding relationships, like placing a bread slice inside a toaster.
  • Attentive Object-Model: This is the novel auxiliary task. Instead of just predicting the next state, it frames learning object representations as a classification problem. It uses random object embeddings as "incorrect labels" to train a model that predicts how object representations change after an interaction. This forces the model to learn meaningful object properties and relationships. Crucially, this object-model also leverages the inter-object attention mechanism, allowing it to learn from the relationships between objects.

The combined objective is to minimize both the standard reinforcement learning loss (for task performance) and the loss from the attentive object-model (for representation learning).

Datasets

The experiments were conducted in the AI2Thor simulated kitchen environment. The authors also created a dataset of both programmatically generated and random object interactions within AI2Thor to train and evaluate their object-representation learning methods.

Key Insights and Ideas

  • Learning object representations is key for sparse rewards: The paper strongly suggests that the bottleneck in learning complex object-interaction tasks with sparse rewards is the agent's ability to learn good representations of objects and their relationships.
  • Auxiliary tasks can bridge the supervision gap: By framing object-model learning as a classification problem with cleverly chosen "incorrect" labels, the authors create a dense learning signal that significantly boosts performance.
  • Attention is crucial for relational understanding: Incorporating inter-object attention into both the policy and the object-model is vital for capturing the complex relationships between objects, which are essential for many tasks.
  • Object properties and relations matter more than just categories: The experiments show that learning to represent object properties (e.g., "cooked") and relations (e.g., "inside") is more critical for task success than just recognizing object categories, especially in challenging tasks.

Key Results

  • LOAD achieved performance closest to an "oracle" agent (which had access to ground-truth object information) in terms of both learning speed and maximum success rate across several challenging tasks.
  • LOAD significantly improved sample efficiency compared to baselines that lacked the attentive object-model, closing the performance gap to the oracle agent by up to 6/8 tasks.
  • Ablation studies showed that removing inter-object attention from the object-model significantly degraded performance, highlighting its importance.
  • The learned object-model was shown to capture object-category, object-properties, and containment relationships significantly better than other unsupervised representation learning methods.

Limitations and Future Work

  • Simulated Environment: The current work is demonstrated in a simulated environment (AI2Thor). Transferring these learned skills to real-world robots remains a challenge.
  • Task Complexity: While the tasks are complex, they are still within a defined set. Generalizing to entirely novel and more complex tasks might require further advancements.
  • Future Work: The authors suggest that their approach is a promising step towards agents that can efficiently learn complex object-interaction tasks. Future work could explore applying this to more diverse environments and tasks, and investigating more sophisticated object-modeling techniques.

Summary updated

Abstract

First-person object-interaction tasks in high-fidelity, 3D, simulated environments such as the AI2Thor virtual home-environment pose significant sample-efficiency challenges for reinforcement learning (RL) agents learning from sparse task rewards. To alleviate these challenges, prior work has provided extensive supervision via a combination of reward-shaping, ground-truth object-information, and expert demonstrations. In this work, we show that one can learn object-interaction tasks from scratch without supervision by learning an attentive object-model as an auxiliary task during task learning with an object-centric relational RL agent. Our key insight is that learning an object-model that incorporates object-attention into forward prediction provides a dense learning signal for unsupervised representation learning of both objects and their relationships. This, in turn, enables faster policy learning for an object-centric relational RL agent. We demonstrate our agent by introducing a set of challenging object-interaction tasks in the AI2Thor environment where learning with our attentive object-model is key to strong performance. Specifically, we compare our agent and relational RL agents with alternative auxiliary tasks to a relational RL agent equipped with ground-truth object-information, and show that learning with our object-model best closes the performance gap in terms of both learning speed and maximum success rate. Additionally, we find that incorporating object-attention into an object-model's forward predictions is key to learning representations which capture object-category and object-state.
BibTeX
@inproceedings{carvalho2021reinforcement,
  author = {Wilka Carvalho and Anthony Liang and Kimin Lee and Sungryull Sohn and Honglak Lee and Richard L. Lewis and Satinder Singh},
  title = {{Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in First-person Simulated 3D Environments}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/306},
}
Figure