Abstract
We investigate the use of animal videos (observations) to improve Reinforcement Learning (RL) efficiency and performance in navigation tasks with sparse rewards. Motivated by theoretical considerations, we make use of weighted policy optimization for off-policy RL and describe the main challenges when learning from animal videos. We propose solutions and test our ideas on a series of 2D navigation tasks. We show how our methods can leverage animal videos to improve performance over RL algorithms that do not leverage such observations.
Introduction
Reinforcement Learning (RL) is notoriously sample-inefficient, often requiring millions of interactions to master simple tasks, particularly in "sparse reward" environments where feedback is infrequent. While Offline RL aims to mitigate this by using pre-collected datasets, real-world biological data—such as videos of animals—presents unique challenges: missing action/reward labels, significant domain gaps between video and simulation, and unknown behavior policies.
Methodology
To bridge the gap between biological observations and artificial agents, we propose a hybrid framework consisting of three core components:
Adversarial Domain Adaptation (DA)
To address the visual discrepancy between animal videos (source) and the agent's environment (target), we employ a discriminator-based approach. Both domains are mapped into a shared latent embedding space, making features from thermal rodent videos indistinguishable from those of the simulated grid.
Inverse Models for Demonstration Recovery
Since raw videos lack action and reward data, we train inverse models using the agent's online interactions. These models predict the actions and rewards associated with state transitions in the animal videos, effectively "hallucinating" a complete demonstration dataset.
Advantage Weighted Policy Optimization (AWPO)
AWPO is our primary algorithmic contribution. It is an importance-sampling-free (IS-free) off-policy algorithm that treats policy improvement as a weighted supervised learning problem. By bypassing the need to estimate the animal's original policy, AWPO avoids the instabilities typically associated with Importance Sampling in off-policy RL.
Experimental Setup
We evaluated our framework on 2D navigation tasks using three distinct domains:
- Target Domain: A 128x128 RGB grid environment where the agent seeks a goal for a +1 reward.
- Simplified Source Domain: A grid variation used to test the framework under controlled domain shifts.
- Rodent3D Source Domain: Real-world thermal infrared videos of a rodent foraging in an open-field maze, representing a high-complexity domain adaptation challenge.
Key Results
Our experiments demonstrate that agents leveraging animal videos through the AWPO framework significantly outperform pure RL benchmarks.
Key findings include:
- Performance Gains: Agents using offline biological data achieved higher rewards than those relying solely on online interaction (AWPO+GAE/PQL). As seen in the results for the simplified source domain, the inclusion of source data (Ds) provides a substantial boost.
- Necessity of Domain Adaptation: While DA was critical for real-world rodent data, the authors noted that in simplified cases, the discriminator could sometimes be over-sensitive to background differences, making the embedding task harder.
- Imitation with Reward Bonuses: Adding a small constant reward bonus ($r_i$) to the estimated rewards from animal videos encouraged the agent to follow biological demonstrations more closely, aiding exploration in sparse-reward settings.
Conclusion and Future Work
This work demonstrates the feasibility of using biological observations to accelerate RL. By mapping pixels to abstract embeddings and using IS-free optimization, we can extract useful navigation primitives from complex animal behavior. Future research will focus on improving the robustness of domain adaptation and extending the framework to more complex tasks like 3D locomotion and object manipulation.
Summary updated