Impossibly Good Experts and How to Follow Them

Aaron Walsman, Muru Zhang, Sanjiban Choudhury, Dieter Fox, Ali FarhadiICLR 2023 · OpenReview

TL;DR

Standard imitation learning fails when an expert has access to "privileged" information that the learner cannot see, such as a hidden map or perfect sensors. This paper introduces ELF Distillation, a novel framework that uses a "follower" policy to evaluate expert advice and an "explorer" policy to actively seek out the missing information required to make that advice actionable.

What Problem Does This Paper Address?

In the ideal world of Imitation Learning (IL), the teacher and the student see the exact same thing. In the real world, however, experts often have "privileged information." A human driver sees further than a car’s cameras; a synthetic expert in a simulator knows the exact coordinates of every object.

The authors call these "Impossibly Good Experts." If an expert knows the location of a goal and walks straight to it, a learner who doesn't know the location will fail if they simply try to mimic the expert's path. The learner is missing the "exploratory" phase—the part where you look around to find the goal. Standard IL techniques like Behavior Cloning or DAgger often collapse in these settings because they don't account for the information gap; they try to copy the result without learning the search.

Examples of Information Asymmetry

The figure above illustrates various scenarios of information asymmetry. In these examples, a hidden variable $X$ dictates the correct terminal state. The expert knows $X$ immediately at the starting state (A), allowing it to take the optimal path directly. The agent, however, must navigate to specific states to reveal $X$ before it can make the correct choice, highlighting why simply copying the expert's initial actions leads to failure.

Methods

The paper formalizes the conditions under which a learner can (or cannot) recover an optimal policy from a privileged expert. They then propose ELF Distillation (Explorer Learning from Follower).

The core idea is to train two policies simultaneously:

  1. The Follower ($\pi_F$): This policy's sole job is to mimic the expert using standard cross-entropy loss. Crucially, it also learns a value function ($v_F$), which estimates the long-term reward the agent would get if it followed the expert's advice from its current state.
  2. The Explorer ($\pi_L$): This is the policy that actually runs at test time. It is trained using Reinforcement Learning (PPO), but its reward is "shaped" by the Follower’s value function.

By using the Follower's value function as a potential-based reward signal ($r + v_F(s') - v_F(s)$), the Explorer is incentivized to move toward states where the expert’s advice is actually useful. In other words, the Explorer learns to find the "hidden" information (like the location of a key) so that it can eventually act like the expert.

Datasets

The authors evaluated their method on two challenging suites of environments designed to highlight information asymmetry:

  • Minigrid: A series of grid-world tasks (e.g., Early Explore, Backtrack) where the agent must find colored balls to determine which door to open. The expert knows the ball colors in advance; the agent does not.
  • Vizdoom: A 3D first-person environment (Monster Room) where the agent must look through a window to see which exit is safe from a monster. The expert knows the monster's location without looking.

Key Insights and Ideas

  • The Correctness Gap: The authors prove that if an expert's advice doesn't "make sense" given the learner's limited observations, standard IL will never converge to the optimal policy.
  • Expert Advice as a Map, Not a Script: Instead of treating expert actions as the "ground truth" to be copied at every step, ELF treats the expert as a guide that tells you which states are valuable.
  • Value-Based Shaping: The "Aha!" moment is using the value of following the expert to drive exploration. If the Follower's value function is low, the Explorer knows it needs to look elsewhere for information before it can successfully "follow" the teacher.

Key Results

ELF Distillation consistently outperformed strong baselines, including ADVISOR, DAgger, and pure PPO.

  • In the Minigrid "Early Explore" tasks, ELF was often the only method to achieve a perfect success rate, as other methods failed to deviate from the expert's "straight-to-goal" path to find the necessary colored balls.
  • In Vizdoom Monster Room 2, ELF was the only method to make reliable progress. Baselines like ADVISOR tended to "latch" onto the expert's predictable signal without understanding the underlying reason (the monster's location), leading to death when they finally had to make a choice.

Limitations and Future Work

The primary limitation is the bias in the follower's value estimate. Early in training, the Follower hasn't seen enough states to provide an accurate value function. If the Follower provides a poor estimate, it can lead the Explorer into a "dead zone" where it stops exploring useful areas.

The authors suggest that future work could involve more sophisticated training schedules—perhaps training the Follower more heavily in the early stages—or developing methods to automatically detect when the Follower's value function is reliable enough to guide the Explorer.

Summary updated

Abstract

We consider the sequential decision making problem of learning from an expert that has access to more information than the learner. For many problems this extra information will enable the expert to achieve greater long term reward than any policy without this privileged information access. We call these experts ``Impossibly Good'' because no learning algorithm will be able to reproduce their behavior. However, in these settings it is reasonable to attempt to recover the best policy possible given the agent's restricted access to information. We provide a set of necessary criteria on the expert that will allow a learner to recover the optimal policy in the reduced information space from the expert's advice alone. We also provide a new approach called Elf Distillation (Explorer Learning from Follower) that can be used in cases where these criteria are not met and environmental rewards must be taken into account. We show that this algorithm performs better than a variety of strong baselines on a challenging suite of Minigrid and Vizdoom environments.
BibTeX
@inproceedings{walsman2023impossibly,
  author = {Aaron Walsman and Muru Zhang and Sanjiban Choudhury and Dieter Fox and Ali Farhadi},
  title = {{Impossibly Good Experts and How to Follow Them}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=sciA_xgYofB},
}
Figure