Abstract
Reinforcement learning (RL) in environments with sparse rewards remains a significant challenge. While intrinsic motivation methods like curiosity and empowerment have been proposed, they often struggle with environmental noise or indirect causal chains. We introduce ELDEN (Exploration via Local DepENdencies), a novel exploration method that incentivizes agents to discover interactions between entities. ELDEN utilizes partial derivatives of a learned dynamics model to track local dependencies and rewards the agent based on the uncertainty of these dependencies within an ensemble. Our results across four complex domains—including 2D grid worlds and 3D robotic simulations—show that ELDEN effectively solves long-horizon tasks where traditional methods fail.
Introduction
Reinforcement learning struggles in environments where the agent only receives a reward signal at the end of a long sequence of actions. To address this, researchers employ intrinsic rewards to encourage exploration. However, existing methods have notable limitations. Curiosity-based methods, which reward high prediction error, are often distracted by "noisy TVs"—complex but irrelevant physics. Empowerment-based methods, which reward maximal control, focus on direct influence and may ignore critical indirect interactions, such as a stove heating a pot the agent is not touching.
ELDEN addresses these gaps in environments with factored state spaces and chained dependencies. It shifts the focus from predicting how entities move to discovering whether they interact.
Methodology
ELDEN models the environment as a local causal graph where edges represent dependencies between entities.
The core technical innovation lies in using partial derivatives of a learned dynamics model: $| \frac{\partial \hat{p}(s^j_{t+1})}{\partial s^i_t} |$. A high derivative indicates that a change in entity $i$ at time $t$ significantly impacts entity $j$ at $t+1$, signaling a dependency. ELDEN trains an ensemble of these models and uses the variance of predicted dependencies as an intrinsic reward, encouraging the agent to explore states with uncertain interactions. To ensure focus on meaningful interactions, a sparsity regularization term is included to suppress false positives.
Experimental Setup
We evaluated ELDEN on four diverse environments with complex chained dependencies:
- Thawing (Mini-behavior): A 2D grid world task involving a fridge, fish, and sink.
- CarWash (Mini-behavior): A long-horizon task with multiple objects (rag, soap, car).
- Minecraft 2D: A tech-tree environment requiring specific crafting sequences.
- Kitchen (Robosuite): A 3D continuous simulation requiring multi-step cooking tasks.
Results and Discussion
ELDEN significantly outperformed state-of-the-art baselines across almost all domains. In the CarWash, Minecraft 2D, and Kitchen environments, ELDEN was frequently the only method to successfully complete the tasks. Beyond task success, ELDEN achieved superior ROC-AUC scores in identifying ground-truth dependencies compared to attention-based models.
The primary insight is that identifying interaction modes is a more robust exploration signal than perfect trajectory prediction. However, ELDEN may be less effective in tasks requiring extreme precision over interaction discovery. Future work will explore learning low-level motor skills and dependencies simultaneously.
Summary updated