Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation

Zhizhou Ren, Anji Liu, Yitao Liang, Jian Peng, Jianzhu Ma

Abstract

Learning new task-specific skills from a few trials is a fundamental challenge for artificial intelligence. Meta reinforcement learning (meta-RL) tackles this problem by learning transferable policies that support few-shot adaptation to unseen tasks. Despite recent advances in meta-RL, most existing methods require the access to the environmental reward function of new tasks to infer the task objective, which is not realistic in many practical applications. To bridge this gap, we study the problem of few-shot adaptation in the context of human-in-the-loop reinforcement learning. We develop a meta-RL algorithm that enables fast policy adaptation with preference-based feedback. The agent can adapt to new tasks by querying human's preference between behavior trajectories instead of using per-step numeric rewards. By extending techniques from information theory, our approach can design query sequences to maximize the information gain from human interactions while tolerating the inherent error of non-expert human oracle. In experiments, we extensively evaluate our method, Adaptation with Noisy OracLE (ANOLE), on a variety of meta-RL benchmark tasks and demonstrate substantial improvement over baseline algorithms in terms of both feedback efficiency and error tolerance.

Introduction

Standard Meta-Reinforcement Learning (meta-RL) is designed for "few-shot" adaptation, where an agent learns a broad set of skills during training so it can master a new, unseen task with very little data. However, a significant bottleneck remains: most existing meta-RL methods rely on a "dense" numeric reward function to figure out what the new task actually is.

In real-world applications, designing these reward functions is notoriously difficult. For instance, defining a mathematical formula for "cleaning a kitchen" is complex and prone to misalignment. It is much more natural for a human to watch two videos of an agent and provide a binary preference (e.g., "I like trajectory A more than B"). This paradigm, known as preference-based RL, faces two primary hurdles: efficiency and noise. Human time is expensive, requiring adaptation with minimal queries, and human feedback is often inconsistent due to fatigue or misunderstanding.

The ANOLE Framework

The researchers introduce ANOLE (Adaptation with Noisy OracLE), a framework that replaces the standard reward-based inference network with a robust, information-theoretic approach. ANOLE builds on the PEARL (Probabilistic Embeddings for Actor-critic RL) architecture but fundamentally changes how the agent identifies the current task.

Mapping Adaptation to the Rényi-Ulam Game

The core innovation of ANOLE is treating task adaptation as a noisy-channel communication problem. Specifically, the authors map the task-inference problem to the Rényi-Ulam game. In this classical information theory puzzle, a player tries to guess a hidden number by asking "yes/no" questions, while the oracle is permitted to lie a certain number of times.

To solve this game in the context of RL, ANOLE utilizes Berlekamp’s volume, a mathematical tool that quantifies uncertainty while accounting for potential errors. The adaptation process follows a structured search:

  1. Candidate Pool: The agent maintains a pool of possible task embeddings (latent representations of the task).
  2. Optimal Querying: Instead of showing random trajectories, ANOLE uses the Berlekamp volume to select a pair of trajectories that, once compared, will most effectively "shrink" the space of possible tasks.
  3. Robust Adaptation: This turns adaptation into a robust version of binary search, allowing the agent to converge on the correct task even if the human provides incorrect feedback.

Experimental Evaluation

The researchers evaluated ANOLE on six standard MuJoCo-based meta-RL benchmarks, including HalfCheetah (Forward/Backward and Random Target Velocity), Ant (Forward/Backward, Random Target Direction, and Goal location), and Walker2d (Random Target Velocity).

Key Results

  • Superior Efficiency: ANOLE reached near-optimal performance with only 10 preference queries, a significant improvement over traditional methods that require dense rewards.
  • Noise Tolerance: The system is explicitly designed to handle a specific budget of incorrect feedbacks ($K_E$). In tests with a 20% error rate, ANOLE maintained high performance while baseline algorithms like "Greedy Binary Search" collapsed.
  • Non-Parametric Scalability: Unlike PEARL, which uses a neural network to guess the task, ANOLE’s non-parametric search over embeddings allows it to scale better across diverse task types (e.g., switching between velocity and direction tasks).
  • Real-World Validation: In trials with actual human feedback, where the average error rate was approximately 6.2%, ANOLE successfully navigated the noise to master complex tasks like Ant-Rand-Dir using only human clicks.

Discussion and Future Work

While ANOLE demonstrates a significant leap in preference-based adaptation, some limitations remain. The current model assumes static dynamics, meaning only the goal (reward function) changes between tasks, not the physics of the environment. Furthermore, the study simulates preferences by summing ground-truth rewards, whereas human preferences are often "non-Markovian" and care about the specific sequence of events rather than just the total score.

Future research will focus on exploring more expressive partial-ordering systems and ensuring that the query system generates realistic trajectory pairs that do not confuse human oracles. Despite these challenges, ANOLE provides a robust foundation for deploying meta-RL agents in environments where numeric rewards are unavailable and human feedback is the primary source of truth.

Summary updated

Abstract

Learning new task-specific skills from a few trials is a fundamental challenge for artificial intelligence. Meta reinforcement learning (meta-RL) tackles this problem by learning transferable policies that support few-shot adaptation to unseen tasks. Despite recent advances in meta-RL, most existing methods require the access to the environmental reward function of new tasks to infer the task objective, which is not realistic in many practical applications. To bridge this gap, we study the problem of few-shot adaptation in the context of human-in-the-loop reinforcement learning. We develop a meta-RL algorithm that enables fast policy adaptation with preference-based feedback. The agent can adapt to new tasks by querying human's preference between behavior trajectories instead of using per-step numeric rewards. By extending techniques from information theory, our approach can design query sequences to maximize the information gain from human interactions while tolerating the inherent error of non-expert human oracle. In experiments, we extensively evaluate our method, Adaptation with Noisy OracLE (ANOLE), on a variety of meta-RL benchmark tasks and demonstrate substantial improvement over baseline algorithms in terms of both feedback efficiency and error tolerance.
BibTeX
@inproceedings{ren2022efficient,
  author = {Zhizhou Ren and Anji Liu and Yitao Liang and Jian Peng and Jianzhu Ma},
  title = {{Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=61UwgeIotn},
}
Figure