Introduction
Teaching autonomous agents to perform complex, multi-step object-interaction tasks—such as "toasting bread"—in 3D environments remains a significant challenge for reinforcement learning (RL). These tasks are characterized by extremely sparse rewards, where a success signal is only received after a long sequence of correct actions. Furthermore, the high branching factor of possible interactions and the lack of inherent object knowledge (e.g., understanding what a toaster is or how it relates to bread) make random exploration ineffective.
The LOAD Architecture
To address these hurdles, we introduce the Learning Object Attention & Dynamics (LOAD) agent. LOAD is designed to learn object categories, properties, and relationships without ground-truth labels or expert demonstrations.
The architecture consists of two main components:
- Attentive Object-DQN: A relational policy that utilizes an inter-object attention mechanism. This allows the agent to reason about relationships, such as whether a slice of bread is currently contained within a toaster before attempting to toggle the device.
- Attentive Object-Model: An auxiliary task that frames the learning of object dynamics as a contrastive classification problem. Instead of predicting high-dimensional pixel changes, the model learns to identify the correct "next state" embedding of an object from a set of distractors. This provides a dense internal training signal, enabling the agent to learn meaningful representations (e.g., "slicing" changes an "apple" to "apple slices") even in the absence of external rewards.
Experimental Setup
The LOAD agent was evaluated in the AI2Thor environment, a high-fidelity 3D kitchen simulator. The evaluation focused on eight challenging food preparation tasks designed to test visual complexity, multi-object interaction, containment, and state changes.
Key Insights and Results
A core innovation of LOAD is the use of Dynamics as Classification. By matching current object embeddings to their future states, the agent learns robust features related to category, state, and location more efficiently than traditional pixel reconstruction methods. Moreover, the shared attention mechanism ensures that the relational knowledge gained during dynamics training directly informs the RL policy's decision-making.
Key results include:
- Sample Efficiency: LOAD significantly outperformed unsupervised baselines like OCN and COBRA, particularly in tasks involving containment.
- Performance vs. Oracle: LOAD approached the performance of an "Oracle" agent provided with perfect hand-coded object information, achieving nearly 100% of the Oracle's Area Under the Curve (AUC) in several tasks.
- Representation Quality: Learned embeddings predicted ground-truth properties (e.g., "is the toaster on?") with over 98% accuracy, demonstrating the effectiveness of the unsupervised auxiliary task.
Limitations and Future Work
While LOAD excels at handling sparse rewards through internal dynamics, it still requires an initial "self-imitation" buffer of successful episodes, meaning the agent must occasionally succeed through exploration to begin learning. Future work will focus on scaling these interactions to multi-room environments and reducing reliance on pre-trained vision systems for object bounding box proposals.
Summary updated