Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillation

Letian Chen, Rohan R. Paleja, Muyleng Ghuy, Matthew C. Gombolay

Abstract

Reinforcement learning (RL) has achieved tremendous success as a general framework for learning how to make decisions. However, this success relies on the interactive hand-tuning of a reward function by RL experts. On the other hand, inverse reinforcement learning (IRL) seeks to learn a reward function from readily-obtained human demonstrations. Yet, IRL suffers from two major limitations: 1) reward ambiguity - there are an infinite number of possible reward functions that could explain an expert's demonstration and 2) heterogeneity - human experts adopt varying strategies and preferences, which makes learning from multiple demonstrators difficult due to the common assumption that demonstrators seeks to maximize the same reward. In this work, we propose a method to jointly infer a task goal and humans' strategic preferences via network distillation. This approach enables us to distill a robust task reward (addressing reward ambiguity) and to model each strategy's objective (handling heterogeneity). We demonstrate our algorithm can better recover task reward and strategy rewards and imitate the strategies in two simulated tasks and a real-world table tennis task.

Introduction

While Reinforcement Learning (RL) has proven to be a powerful tool for autonomous decision-making, its practical application is often bottlenecked by the need for meticulously hand-crafted reward functions. Poorly designed rewards can lead to "reward hacking," where agents exploit loopholes to achieve high scores without actually fulfilling the intended task. Inverse Reinforcement Learning (IRL) offers a solution by learning reward functions directly from expert demonstrations.

However, standard IRL methods struggle with two primary challenges. First is reward ambiguity, where multiple reward functions can explain the same observed behavior. Second is heterogeneity; humans often employ diverse strategies to solve the same problem. When demonstrators use different styles (e.g., a "safe" vs. a "risky" approach), traditional IRL algorithms often fail by assuming all experts are optimizing a single, identical reward function. This paper introduces Multi-Style Reward Distillation (MSRD) to disentangle the shared task objective from individual strategic preferences.

Methodology

The core innovation of MSRD is the decomposition of the reward function into two distinct components. For any demonstrator $i$, the total reward $R^{(i)}$ is modeled as:

$$R^{(i)} = R^{(0)} + \alpha_i R^{(i)}_s$$

Where:

  • $R^{(0)}$ (Task Reward): The invariant goal shared by all demonstrators (e.g., "win the game").
  • $R^{(i)}_s$ (Strategy Reward): The style-specific preference unique to demonstrator $i$ (e.g., "play aggressively").

To implement this, the researchers utilized a two-column neural network architecture within an adversarial framework based on Adversarial IRL (AIRL). This design allows the model to update the task reward directly from the discriminator's loss, preventing strategy-specific signals from overshadowing the shared task goal. Furthermore, an L2-norm penalty is applied to ensure that while individual strategies may vary, they remain anchored to the primary task objective.

The MSRD architecture for joint task and strategy reward inference.

Experimental Evaluation

The MSRD framework was evaluated across three diverse environments to test its robustness in both simulation and the real world:

  1. Inverted Pendulum (Simulated): A classic control task where the goal is to balance a pole. To simulate heterogeneity, the researchers used the DIAYN (Diversity is All You Need) algorithm to generate 20 distinct balancing styles.
  2. Hopper (Simulated): A MuJoCo environment featuring a one-legged robot. Two distinct styles—"Hop" and "Crawl"—were generated to test strategy recovery.
  3. Robot Table Tennis (Real-World): Using a Sawyer robot arm, 180 trajectories were collected from 10 human participants. Demonstrators kinesthetically taught the robot three specific strokes: push, slice, and topspin, from both forehand and backhand positions.

Results and Discussion

MSRD significantly outperformed the baseline AIRL in recovering the underlying task reward. In the Inverted Pendulum task, MSRD achieved a near-perfect correlation of 0.998 with the ground-truth reward, whereas AIRL only reached a correlation of 0.513. This demonstrates that by seeing multiple ways to solve a task, MSRD can more effectively identify the invariant task goal.

Comparison of task reward recovery between MSRD and the baseline AIRL.

Beyond task recovery, MSRD successfully captured stylistic nuances. In the Hopper environment, the learned "Hop" strategy reward was significantly higher when the robot hopped than when it crawled, confirming the model's ability to isolate style. In the real-world table tennis task, the robot achieved an 83% success rate in returning balls after only 30 minutes of training per strategy, successfully mimicking the specific spins taught by human demonstrators.

Conclusion

Multi-Style Reward Distillation provides a robust framework for learning from diverse, heterogeneous demonstrations. By disentangling the "what" (task goal) from the "how" (strategy), MSRD addresses the fundamental IRL challenges of reward ambiguity and demonstrator variety. Future work may explore unsupervised clustering to infer strategy labels automatically and investigate non-linear combinations of task and strategy rewards to model even more complex human behaviors.

Summary updated

Abstract

Reinforcement learning (RL) has achieved tremendous success as a general framework for learning how to make decisions. However, this success relies on the interactive hand-tuning of a reward function by RL experts. On the other hand, inverse reinforcement learning (IRL) seeks to learn a reward function from readily-obtained human demonstrations. Yet, IRL suffers from two major limitations: 1) reward ambiguity - there are an infinite number of possible reward functions that could explain an expert's demonstration and 2) heterogeneity - human experts adopt varying strategies and preferences, which makes learning from multiple demonstrators difficult due to the common assumption that demonstrators seeks to maximize the same reward. In this work, we propose a method to jointly infer a task goal and humans' strategic preferences via network distillation. This approach enables us to distill a robust task reward (addressing reward ambiguity) and to model each strategy's objective (handling heterogeneity). We demonstrate our algorithm can better recover task reward and strategy rewards and imitate the strategies in two simulated tasks and a real-world table tennis task.
BibTeX
@inproceedings{chen2020joint,
  author = {Letian Chen and Rohan R. Paleja and Muyleng Ghuy and Matthew C. Gombolay},
  title = {{Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillation}},
  booktitle = {HRI},
  year = {2020},
  url = {https://openreview.net/forum?id=dzhDzBEXyk},
}
Figure