Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment

Wilka Carvalho, Anthony Liang, Kimin Lee, Sungryull Sohn, Honglak Lee, Richard L. Lewis, Satinder Singh

Introduction

Teaching autonomous agents to perform complex, multi-step object-interaction tasks—such as "toasting bread"—in 3D environments remains a significant challenge for reinforcement learning (RL). These tasks are characterized by extremely sparse rewards, where a success signal is only received after a long sequence of correct actions. Furthermore, the high branching factor of possible interactions and the lack of inherent object knowledge (e.g., understanding what a toaster is or how it relates to bread) make random exploration ineffective.

The LOAD Architecture

To address these hurdles, we introduce the Learning Object Attention & Dynamics (LOAD) agent. LOAD is designed to learn object categories, properties, and relationships without ground-truth labels or expert demonstrations.

The LOAD agent architecture, featuring an attentive policy and a contrastive dynamics model.

The architecture consists of two main components:

  1. Attentive Object-DQN: A relational policy that utilizes an inter-object attention mechanism. This allows the agent to reason about relationships, such as whether a slice of bread is currently contained within a toaster before attempting to toggle the device.
  2. Attentive Object-Model: An auxiliary task that frames the learning of object dynamics as a contrastive classification problem. Instead of predicting high-dimensional pixel changes, the model learns to identify the correct "next state" embedding of an object from a set of distractors. This provides a dense internal training signal, enabling the agent to learn meaningful representations (e.g., "slicing" changes an "apple" to "apple slices") even in the absence of external rewards.

Experimental Setup

The LOAD agent was evaluated in the AI2Thor environment, a high-fidelity 3D kitchen simulator. The evaluation focused on eight challenging food preparation tasks designed to test visual complexity, multi-object interaction, containment, and state changes.

Sequential frames from a task in the AI2Thor environment, demonstrating state changes and object interactions.

Key Insights and Results

A core innovation of LOAD is the use of Dynamics as Classification. By matching current object embeddings to their future states, the agent learns robust features related to category, state, and location more efficiently than traditional pixel reconstruction methods. Moreover, the shared attention mechanism ensures that the relational knowledge gained during dynamics training directly informs the RL policy's decision-making.

Key results include:

  • Sample Efficiency: LOAD significantly outperformed unsupervised baselines like OCN and COBRA, particularly in tasks involving containment.
  • Performance vs. Oracle: LOAD approached the performance of an "Oracle" agent provided with perfect hand-coded object information, achieving nearly 100% of the Oracle's Area Under the Curve (AUC) in several tasks.
  • Representation Quality: Learned embeddings predicted ground-truth properties (e.g., "is the toaster on?") with over 98% accuracy, demonstrating the effectiveness of the unsupervised auxiliary task.

Limitations and Future Work

While LOAD excels at handling sparse rewards through internal dynamics, it still requires an initial "self-imitation" buffer of successful episodes, meaning the agent must occasionally succeed through exploration to begin learning. Future work will focus on scaling these interactions to multi-room environments and reducing reliance on pre-trained vision systems for object bounding box proposals.

Summary updated

Abstract

Learning how to execute complex tasks involving multiple objects in a 3D world is challenging when there is no ground-truth information about the objects or any demonstration to learn from. When an agent only receives a signal from task-completion, this makes it challenging to learn the object-representations which support learning the correct object-interactions needed to complete the task. In this work, we formulate learning an attentive object dynamics model as a classification problem, using random object-images to define incorrect labels for our object-dynamics model. We show empirically that this enables object-representation learning that captures an object's category (is it a toaster?), its properties (is it on?), and object-relations (is something inside of it?). With this, our core learner (a relational RL agent) receives the dense training signal it needs to rapidly learn object-interaction tasks. We demonstrate results in the 3D AI2Thor simulated kitchen environment with a range of challenging food preparation tasks. We compare our method's performance to several related approaches and against the performance of an oracle: an agent that is supplied with ground-truth information about objects in the scene. We find that our agent achieves performance closest to the oracle in terms of both learning speed and maximum success rate.
BibTeX
@inproceedings{carvalho2021reinforcement,
  author = {Wilka Carvalho and Anthony Liang and Kimin Lee and Sungryull Sohn and Honglak Lee and Richard L. Lewis and Satinder Singh},
  title = {{Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/306},
}
Figure