Overview
This paper introduces Meta Reward Learning (MeRL) and MAPOX, a two-pronged approach designed to help Reinforcement Learning (RL) agents succeed in environments where feedback is both rare and potentially misleading. By using meta-learning to refine reward functions based on validation performance, and combining mode-covering exploration with mode-seeking optimization, the authors achieve state-of-the-art results on complex semantic parsing tasks like WikiSQL and WikiTableQuestions.
The Challenge: Sparse and Underspecified Rewards
In many real-world AI tasks—such as teaching a computer to write SQL code from a natural language question—the agent only receives a binary "success" or "failure" signal. This creates two massive hurdles for standard RL algorithms:
- Sparse Rewards: In a massive search space of possible actions or code sequences, the agent might almost never stumble upon a successful sequence purely by chance, making initial learning nearly impossible.
- Underspecified Rewards: Even when the agent succeeds, it might be for the wrong reason. For example, if asked "Which nation won the most silver medals?", an agent might accidentally write a program that looks for the most gold medals. If the same country happens to have the most of both in one specific table, the agent gets a "success" reward for a logically flawed program.
This phenomenon is known as a spurious trajectory. It prevents the agent from generalizing to new data where that "lucky" coincidence doesn't hold.
Proposed Framework: MAPOX and MeRL
The authors propose a framework that explicitly decouples exploration from robust policy training, utilizing a meta-learning loop to correct for spurious rewards.
1. MAPOX (MAPO eXploratory)
Standard RL objectives are often "mode-seeking," meaning they focus heavily on exploiting the best path found so far. However, for initial exploration, "mode-covering" objectives like Iterative Maximum Likelihood (IML) are superior because they encourage the agent to find any and all successful paths.
MAPOX leverages this by using IML to discover a diverse set of successful trajectories to fill a memory buffer. Once a rich set of trajectories is found, the system switches to a robust objective (MAPO) to train the final policy.
2. Meta Reward Learning (MeRL)
To solve the "spurious success" problem, the authors learn an auxiliary reward function that operates in a dual-loop structure:
- The Inner Loop: The agent trains its policy using a combination of the environment's standard binary reward and an auxiliary reward (based on features like program length or entity matches).
- The Outer Loop (Meta-Step): The parameters of that auxiliary reward are optimized to maximize the agent's performance on a held-out validation set.
The underlying logic is elegant: if a reward function encourages spurious programs, the agent will inevitably fail on the validation set. Therefore, the meta-optimizer learns to "discount" those misleading rewards, forcing the agent to focus on programs that truly capture the underlying logic. The authors also tested BoRL (Bayesian Optimization Reward Learning), a gradient-free alternative for cases where the validation objective isn't differentiable.
Key Insights
- Generalization as a Signal: The most powerful idea here is using validation performance as a proxy for "correctness." Since we lack "gold" programs (only the final answers are known), we cannot tell the agent exactly why it was right. However, we can observe if its logic holds up on new data.
- IML vs. RER: The paper clarifies a debate in the field: IML (Maximum Likelihood) is superior for exploration because it covers all successful modes, whereas RER (Regularized Expected Return) is better for exploitation. MAPOX combines them to get the best of both worlds.
- Automatic Feature Weighting: Manually tuning reward weights (e.g., "how much should I penalize long programs?") is tedious and error-prone. MeRL automates this by treating these weights as meta-parameters.
Datasets and Results
The approach was evaluated on three benchmarks, setting new state-of-the-art records for weakly-supervised semantic parsing at the time of publication:
- WikiTableQuestions: A complex semantic parsing task requiring reasoning over Wikipedia tables. MeRL improved accuracy to 44.1%, a 1.2% absolute gain over the previous best.
- WikiSQL: A large-scale dataset for mapping natural language questions to SQL queries. MeRL achieved 74.8% accuracy (a 2.4% gain). When using an ensemble of 5 models, this rose to 76.9%.
- Instruction Following Maze: A synthetic environment designed specifically to test an agent's ability to ignore "deadly traps" and follow instructions. MeRL significantly outperformed standard RL, effectively ignoring spurious paths that led to the goal without following the actual instructions.
Limitations and Future Work
While highly effective, the current implementation of the auxiliary reward relies on a linear combination of hand-engineered features. The authors suggest that using more powerful neural network reward models could capture more subtle nuances of "spuriousness." Additionally, gradient-based meta-learning can get stuck in local optima, suggesting a need for random restarts or sophisticated annealing schedules. Finally, extending this framework from terminal rewards to "dense" rewards (feedback at every step) remains a promising avenue for future research.
Summary updated