First-Explore, then Exploit: Meta-Learning Intelligent Exploration

Ben Norman, Jeff CluneSubmission to ICLR 2024 · Withdrawn

TL;DR

Reinforcement learning agents typically struggle in environments where finding the optimal path requires taking actions that hurt short-term rewards. First-Explore solves this by decoupling meta-reinforcement learning into two dedicated policies: one trained solely to gather information ("sacrificial exploration"), and another trained purely to maximize return using that context ("exploitation"). By grounding exploration directly in how much it boosts subsequent exploitation, First-Explore avoids local optima and dramatically outperforms standard RL and meta-RL baselines in deceptive environments.

What Problem Does This Paper Address?

Standard RL and meta-RL agents usually rely on a single policy to handle both exploration (gathering data) and exploitation (maximizing reward). In practice, this means exploration is largely coincidental—the agent exploits while injecting noise (e.g., via entropy bonuses or $\epsilon$-greedy actions).

This strategy breaks down when environments feature deceptive or hostile reward landscapes, where discovering high-value states requires taking immediate, negative-reward penalties (e.g., enduring pain to learn a skill, or paying a fee to unlock a map). Because cumulative-reward meta-RL methods ($RL^2$, VariBAD, HyperX) penalize any drop in current returns, they often learn to "freeze" or do nothing to avoid penalties, failing to ever discover high rewards.

Methods

First-Explore resolves this tension by training two distinct policies that share context:

  • The Explore Policy ($\pi_{\text{explore}}$): Its sole job is to gather informative rollouts. Crucially, its training objective is grounded directly in the exploit policy: an explore rollout is rewarded based on how much it improves the return of the subsequent exploit rollout.
  • The Exploit Policy ($\pi_{\text{exploit}}$): Conditioned on the history of all preceding exploration rollouts, it strictly attempts to maximize reward without needing to explore.
Explore (τ₁) ──> Exploit (τ'₁) [Evaluates info gain]
     │
     ▼
Explore (τ₂) ──> Exploit (τ'₂)

Architecture and Training

  • Transformer Backbone: Uses a GPT-2-style decoder-only transformer implemented in JAX. Both policies share the underlying transformer representations of the historical rollout context, differing only via separate final linear prediction heads.
  • Conditional Optimization Loss: Rather than standard policy gradients, the authors employ an imitation-style cross-entropy objective. The exploit policy is trained on trajectories that meet or exceed the maximum reward seen so far in context, while the explore policy trains on actions that produce informative context leading to higher subsequent exploit rewards.

Datasets & Environments

The authors evaluate First-Explore on synthetic benchmark tasks designed to test sacrificial exploration:

  • 10-Armed Gaussian Bandit: Arm means sampled from $\mathcal{N}(0, I)$ with Gaussian noise $\sigma = 0.5$. Tested against UCB1, Thompson Sampling, $RL^2$, and round-robin baselines.
  • $9 \times 9$ Dark Treasure-Room (Hostile): A gridworld inspired by Darkroom (Laskin et al., 2022). The agent only observes its coordinates $(x, y)$. Eight hidden items are placed uniformly at random with rewards $r \sim U[-4, 2]$. Because the expected reward of entering an unknown cell is negative ($-1.0$), blind exploration is punishing.
  • Kind-Room Domain (Control): Structurally identical to the Dark Treasure-Room, but object rewards are strictly positive ($r \sim U[0, 2]$), testing behavior when coincidental exploration is sufficient.

Key Insights and Ideas

  • Sacrificial Exploration Unlocked: Unlike intrinsic motivation (e.g., novelty bonuses) which often succumbs to the "noisy TV" problem by chasing useless randomness, First-Explore grounds exploration directly in downstream utility. It explores only what helps future exploitation.
  • Decoupling Solves Baselines Freezing: In the hostile Dark Treasure-Room, cumulative-reward baselines ($RL^2$, VariBAD) learned to stay completely stationary to avoid negative traps. First-Explore freely absorbed early penalties to map the space.
  • Higher Cumulative Reward via Sacrifice: Even though First-Explore's explore phase ignores episode returns, running two explore episodes followed by exploitation yielded higher cumulative reward than meta-RL baselines explicitly optimized for cumulative reward across episodes.

Key Results

  • 10-Armed Bandit:
  • First-Explore learned an optimal exhaustive search strategy, pulling all 10 distinct arms within the first 10 steps (matching round-robin coverage), whereas $RL^2$ and Thompson Sampling averaged $< 8$ unique arms covered.
  • Once coverage was established, its exploit policy reached near-oracle performance ($\approx 1.55$ average pull reward), outperforming all baseline policies.
  • Dark Treasure-Room:
  • Baselines catastrophically failed: $RL^2$ and VariBAD averaged $< 2$ unique coordinates visited; HyperX visited fewer states than random action selection. All hovered near $0$ cumulative reward.
  • First-Explore consistently visited 9 unique coordinates in episode 1 and 8 more in episode 2, translating to a high cumulative reward ($\approx 2.0$ at step 90).
  • Kind-Room: Even when rewards were purely positive and baselines did not freeze, First-Explore's exploitation policy still outperformed all competitors.

Limitations and Future Work

  • Myopic Exploration: The explore policy is trained greedily against the immediate next exploit episode rather than planned across a multi-episode horizon, which can lead to suboptimal long-term exploration sequences.
  • Safety Concerns in Real Environments: A willingness to accept arbitrary negative rewards during exploration is dangerous in physical settings (e.g., robotics), where negative rewards correspond to hardware damage or safety violations.
  • Scaling Long Sequences: Conditioning on extensive multi-episode interaction histories demands substantial transformer context lengths and GPU compute.

Summary updated

Abstract

Standard reinforcement learning (RL) agents never intelligently explore like a human (i.e. by taking into account complex domain priors and previous explorations). Even the most basic intelligent exploration strategies such as exhaustive search are only inefficiently or poorly approximated by approaches such as novelty search or intrinsic motivation, let alone more complicated strategies like learning new skills, climbing stairs, opening doors, or conducting experiments. This lack of intelligent exploration limits sample efficiency and prevents solving hard exploration domains. We argue a core barrier prohibiting many RL approaches from learning intelligent exploration is that the methods attempt to explore and exploit simultaneously, which harms both exploration and exploitation as the goals often conflict. We propose a novel meta-RL framework (First-Explore) with two policies: one policy learns to only explore and one policy learns to only exploit. Once trained, we can then explore with the explore policy, for as long as desired, and then exploit based on all the information gained during exploration. This approach avoids the conflict of trying to do both exploration and exploitation at once. We demonstrate that First-Explore can learn intelligent exploration strategies such as exhaustive search and more, and that it outperforms dominant standard RL and meta-RL approaches on domains where exploration requires sacrificing reward. Surprisingly and importantly, on such domains, First-Explore not only achieves higher final episode reward, it also achieves higher cumulative reward. First-Explore is a significant step towards creating meta-RL algorithms capable of learning human-level exploration, which is essential to solve challenging unseen hard-exploration domains.
BibTeX
@inproceedings{norman2024first,
  author = {Ben Norman and Jeff Clune},
  title = {{First-Explore, then Exploit: Meta-Learning Intelligent Exploration}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=AhjTu2aiiW},
}
Figure