Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation

Zhizhou Ren, Anji Liu, Yitao Liang, Jian Peng, Jianzhu Ma

TL;DR

Meta-Reinforcement Learning (meta-RL) typically requires precise, per-step numeric rewards to adapt to new tasks—a condition rarely met in real-world human-robot interactions. ANOLE (Adaptation with Noisy OracLE) is a novel algorithm that enables agents to adapt to new tasks using only a handful of binary human preferences (e.g., "I prefer behavior A over B"). Crucially, ANOLE remains robust to the inevitable mistakes and inconsistencies inherent in human feedback.

The Challenge: Moving Beyond Perfect Rewards

The core objective of meta-RL is "learning to learn": training an agent across a variety of tasks so it can adapt to a brand-new, unseen task with minimal samples. However, most existing meta-RL algorithms, such as PEARL, rely heavily on receiving a dense numeric reward signal for every single step during the adaptation phase.

In practical applications, defining a perfect reward function is notoriously difficult. It is much more natural for a human user to observe two agent behaviors and indicate which one looks better. This approach, known as preference-based RL, introduces its own challenges. Preference signals are "weak" (providing just one bit of information per trajectory comparison), and human oracles are "noisy" (prone to mistakes or changing their minds). ANOLE bridges this gap, providing a mathematically grounded way to perform few-shot adaptation using sparse, potentially incorrect human comparisons.

Methods: Information Theory Meets Meta-RL

ANOLE builds upon the PEARL framework, which represents different tasks as points in a low-dimensional latent task embedding space. To handle the noise and sparsity of human feedback, the researchers drew a clever connection between task adaptation and an information-theory classic: Rényi-Ulam’s game. In this game, one player thinks of a number, and the other tries to guess it by asking "yes/no" questions, knowing the first player might lie a certain number of times.

To implement this concept, ANOLE introduces several key components:

  • Preference Predictor: A model trained during the meta-training phase to predict which trajectory an oracle would prefer, given a specific latent task embedding.
  • Berlekamp’s Volume: This mathematical tool measures uncertainty in a noisy communication channel. ANOLE uses it to quantify how much "volume" (uncertainty) remains in the task space given the history of noisy feedback.
  • Active Query Strategy: Instead of showing the human random trajectories, ANOLE uses a binary-search-like strategy. It selects pairs of trajectories that will reduce the Berlekamp volume as much as possible in the worst-case scenario, regardless of the human's answer.

The overall adaptation loop is summarized in the algorithm below:

Algorithm 1: Adaptation with Noisy OracLE (ANOLE)

Datasets and Evaluation

The authors evaluated ANOLE on six standard meta-RL benchmarks based on the MuJoCo physics engine:

  • HalfCheetah: Forward/Backward and Random Target Velocity.
  • Ant: Forward/Backward, Random Target Direction, and Random Target Goal.
  • Walker2d: Random Target Velocity.

Beyond simulated oracles, they also conducted experiments with real human feedback. Participants compared 2D projections of agent trajectories, evaluating the algorithm's performance in a truly "in-the-loop" setting.

Key Insights

  • Robustness through Information Theory: By treating human feedback as a "noisy channel," the authors utilized established error-correction mathematics to handle mistakes. This provides a more principled and robust approach than simply trying to "average out" noise.
  • Non-parametric Inference: Standard meta-RL uses a neural network encoder to guess the task. ANOLE instead uses a search-based approach over a pool of candidate task embeddings. This proves much more effective when feedback is sparse (e.g., only 10 comparisons) and binary.
  • Query Design Matters: The paper demonstrates that what you ask the human is just as important as how you process the answer. By picking "balanced" queries that split the task space roughly in half, the agent learns exponentially faster.

Key Results

ANOLE demonstrated significant improvements over baselines like PEARL and simple greedy search, particularly under noisy feedback conditions.

  • Efficiency: ANOLE reached high performance levels with as few as 10 preference queries.
  • Error Tolerance: In simulations with a 20% error rate (where the oracle gives the wrong answer 1 out of 5 times), ANOLE maintained high performance while baseline success rates plummeted.
  • Human-in-the-loop: In real human trials (which exhibited an error rate of about 6.2%), ANOLE successfully adapted to tasks like "Ant-Rand-Dir" and "HalfCheetah-Fwd-Back" using imperfect human judgments.

As seen in the learning curves below, ANOLE consistently approaches the performance of agents that have access to perfect, dense rewards, far outperforming standard preference-based adaptations.

Learning curves across various MuJoCo tasks. ANOLE (orange) significantly outperforms baselines in adaptation speed and final return.

Limitations and Future Work

While ANOLE presents a significant step forward, the authors note a few limitations:

  • Static Dynamics: The current model assumes the physics of the environment remain constant, with only the goal (reward) changing. Adapting to changing dynamics (e.g., a robot with a broken leg) is a challenge for future work.
  • Linear Reward Assumption: The feedback is currently simulated by summing up rewards. However, human preferences can be more complex and non-linear. Future iterations could better exploit the expressiveness of partial orderings.
  • Unrealistic Queries: Sometimes the mathematically "most informative" query might involve a trajectory that looks physically bizarre, potentially confusing a human observer. Improving the "human-friendliness" of the queries remains an important area for improvement.

Summary updated

Abstract

Learning new task-specific skills from a few trials is a fundamental challenge for artificial intelligence. Meta reinforcement learning (meta-RL) tackles this problem by learning transferable policies that support few-shot adaptation to unseen tasks. Despite recent advances in meta-RL, most existing methods require the access to the environmental reward function of new tasks to infer the task objective, which is not realistic in many practical applications. To bridge this gap, we study the problem of few-shot adaptation in the context of human-in-the-loop reinforcement learning. We develop a meta-RL algorithm that enables fast policy adaptation with preference-based feedback. The agent can adapt to new tasks by querying human's preference between behavior trajectories instead of using per-step numeric rewards. By extending techniques from information theory, our approach can design query sequences to maximize the information gain from human interactions while tolerating the inherent error of non-expert human oracle. In experiments, we extensively evaluate our method, Adaptation with Noisy OracLE (ANOLE), on a variety of meta-RL benchmark tasks and demonstrate substantial improvement over baseline algorithms in terms of both feedback efficiency and error tolerance.
BibTeX
@inproceedings{ren2022efficient,
  author = {Zhizhou Ren and Anji Liu and Yitao Liang and Jian Peng and Jianzhu Ma},
  title = {{Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=61UwgeIotn},
}
Figure