PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning

Angelos Filos, Clare Lyle, Yarin Gal, Sergey Levine, Natasha Jaques, Gregory Farquhar

In the real world, autonomous agents—like self-driving cars or robots—rarely act in isolation. They are surrounded by other agents pursuing their own varied goals. Traditional Reinforcement Learning (RL) usually ignores this rich source of observational data, while Imitation Learning (IL) often fails because it assumes the demonstrator is an expert pursuing the exact same goal as the learner.

$\Psi\Phi$-learning (pronounced "Sci-Fi") is a novel reinforcement learning framework designed to learn from the unlabeled, reward-free demonstrations of other agents. By disentangling environment dynamics from agent-specific goals, it can "borrow" useful behaviors from others even when their objectives differ from its own, while gracefully falling back to standard RL when demonstrations are irrelevant or misleading.

The Challenge of No-Reward Demonstrations

The authors tackle the "no-reward demonstration" setting: an agent observes what others are doing, but lacks access to their rewards, expertise levels, or specific destinations. For instance, if a self-driving car is trying to reach a highway exit, how can it utilize data from a nearby driver who is searching for a parking spot? Standard imitation methods might mistakenly lead the ego-vehicle to park. $\Psi\Phi$-learning aims to extract the general "how-to-drive" knowledge without blindly copying the "where-to-go" objective.

Methodology: Disentangling Dynamics and Preferences

The core of $\Psi\Phi$-learning relies on the disentanglement of Successor Features (SFs) and Preferences.

  • Successor Features ($\Psi$): These represent the expected future state features an agent will encounter, capturing "what happens" if a certain policy is followed.
  • Preferences ($w$): A vector representing "what the agent wants." The reward is simply the dot product of these two components: $R = \Phi^\top w$.

To leverage this structure, the authors introduce two major algorithmic contributions:

  1. Inverse Temporal Difference (ITD) Learning: An offline multi-task Inverse RL algorithm. It observes demonstrations and "explains" them by learning shared state features (cumulants $\Phi$), per-agent successor features ($\Psi^k$), and preference vectors ($w^k$). It uses Temporal Difference (TD) errors to find the features that make the observed behavior look rational.
  2. $\Psi\Phi$-Learning: This combines ITD with online RL. The "ego-agent" uses Generalized Policy Improvement (GPI) to evaluate the policies of other agents on its own task. If another agent's strategy is currently useful, the ego-agent adopts it; if not, it relies on its own learned policy. Crucially, both the imitation and the RL components share the same underlying state features ($\Phi$), creating a powerful representation learning loop.
The $\Psi\Phi$-learning architecture: ITD recovers others' features, while GPI selects the best policy for the ego-task.

Evaluation Environments

The researchers evaluated their method across three distinct environments to test different capabilities:

  • Highway: A multi-agent autonomous driving simulator. Scenarios included "single-task" (everyone has the same goal), "adversarial" (others behave poorly), and "multi-task" (the most realistic setting with diverse driving styles).
  • CoinGrid: A grid-world environment used to test transfer learning. Agents collect different colored coins, allowing for clear visualization of learned preferences.
  • FruitBot: A high-dimensional, pixel-based environment from the OpenAI ProcGen suite. This tested the algorithm's ability to scale to complex visual inputs and deep neural architectures.
Visualizations of the environments used for evaluation: Highway, Roundabout, CoinGrid, and FruitBot.

Key Insights and Results

By separating "how the world works" from "what I want," the agent can learn from anyone. It doesn't matter if a demonstrator is suboptimal or has a different goal; the agent can still extract useful information about environment dynamics. Using the same state features ($\Phi$) for both modeling others and learning for oneself allows for a "transfer of salience." Learning what is important to others helps the ego-agent understand what features of the environment are generally important.

Empirical results highlight several major advantages of this approach:

  • Superior Multi-Task Performance: In the realistic multi-task Highway scenario, $\Psi\Phi$-learning significantly outperformed baselines like DQN, GAIL, and SQIL.
  • Robustness to Bad Data (Graceful Fallback): Unlike pure imitation learning, which can be distracted by irrelevant demonstrations, $\Psi\Phi$-learning defaults to standard RL performance in the worst-case scenario. In "adversarial" scenarios where other agents were useless, $\Psi\Phi$-learning maintained performance while imitation-heavy baselines crashed.
  • Effective IRL: ITD proved to be a highly effective Inverse RL method, often outperforming GAIL in recovering reward functions from demonstrations.
  • Zero-Shot Task Inference and Rapid Adaptation: Because the agent learns the space of possible goals, it can adapt to a brand-new task almost instantly. By observing just a few rewards in a new setting, it can solve a least-squares problem to infer its new preference vector ($w$) and immediately apply GPI. In CoinGrid transfer tasks, the agent achieved near-optimal performance in just 0 to 1 episodes, whereas baselines like SQIL required up to 100 episodes to catch up.

Limitations and Future Work

While promising, the authors identify a few areas for improvement. Currently, the ITD component scales linearly with the number of distinct demonstrators. Using Universal Successor Features could handle an arbitrary number of agents more efficiently. Additionally, the current model treats others as part of the environment dynamics and does not yet account for "recursive reasoning" (e.g., "I know that you know that I'm going to change lanes"). Future work could explore adapting the framework to strategic settings where high-level coordination and opponent modeling are essential.

Summary updated

Abstract

We study reinforcement learning (RL) with no-reward demonstrations, a setting in which an RL agent has access to additional data from the interaction of other agents with the same environment. Howe...
BibTeX
@inproceedings{filos2021psiphi,
  author = {Angelos Filos and Clare Lyle and Yarin Gal and Sergey Levine and Natasha Jaques and Gregory Farquhar},
  title = {{PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=eISmeNSosF},
}
Figure