The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewards

Sukai Huang, Shu-Wei Liu, Nir Lipovetzky, Trevor Cohn

TL;DR

Using Vision-Language Models (VLMs) to generate reward signals for reinforcement learning agents sounds promising, but reward noise often cripples their performance. This paper demonstrates that false positive rewards (rewarding incorrect behavior) are uniquely destructive to agent learning and introduces BiMI (Binary Mutual Information), a novel reward function that eliminates false positives to drastically improve training efficiency in complex environments.

What Problem Does This Paper Address?

Reinforcement learning (RL) in sparse-reward environments—like complex video games or robotic navigation tasks—is notoriously difficult because agents rarely stumble upon the goal by chance. To guide them, a popular modern approach uses pre-trained Vision-Language Models (VLMs) to compute a "similarity score" (usually via cosine similarity) between the agent's visual trajectory and a natural language instruction (e.g., "grab the key").

Figure 1: Illustration of embodied RL agents using VLM reward model

However, the authors find a major catch: agents trained with VLM rewards frequently underperform compared to agents using simple, instruction-blind intrinsic exploration rewards. The culprit is reward noise. Specifically, standard cosine similarity metrics suffer from two major flaws:

  1. State Entanglement: The model rewards textually or visually similar states that lead to completely different outcomes (e.g., confusing "take the red pill" with "take the blue pill").
  2. Composition Insensitivity: The model fails to account for the correct order of multi-step tasks or gives high rewards for incomplete trajectories.

This creates rampant false positive rewards, leading to "reward hacking" where the agent learns to exploit the VLM's blind spots rather than actually solving the task.

Methods

The authors frame VLM rewards through the lens of Heuristic-Guided RL (HuRL). Mathematically, they prove that while false negatives (failing to reward a correct action) merely slow down learning, false positives (rewarding a wrong action) break "Bellman-consistent pessimism," causing an unbounded increase in the optimality gap and severely stalling convergence.

To fix this, they propose BiMI (Binary Mutual Information Reward), which introduces two key mechanisms:

  • Binary Signal (BI): Instead of providing continuous rewards based on raw cosine similarity, BiMI uses conformal prediction on a calibration set to establish a strict confidence threshold ($\hat{q}$). The agent receives a strict, one-time binary reward (+1) only when the VLM is highly confident the instruction was met. This acts as a pessimistic filter.
  • Mutual Information (MI) Maximization: To prevent reward hacking on frequently repeated actions, BiMI adds a regularization term that penalizes rewards from instructions that occur too often in the environment.

The final reward function is defined as: $$r_{\text{BiMI}}^v(\tau, l_k) = \max(\mathbf{1}_{\{p(l_k|\tau) \ge \hat{q}\}} - p(l_k), 0)$$

Datasets

The authors evaluated their approach across three challenging, long-horizon, sparse-reward environments:

  • Montezuma's Revenge (Atari): A notoriously difficult platformer requiring precise sequencing. The authors manually annotated 97 sequential instructions.
  • Crafter: An open-world 2D survival game inspired by Minecraft, modified to feature a linear sequence of instructions leading to mining a diamond.
  • Minigrid 'Go To Seq': A grid-world maze requiring long-horizon navigation and object interactions in a specific order.

They used CLIP and X-CLIP as VLM backbones, fine-tuned on environment-specific expert trajectories (~80,000 pairs for Crafter/Minigrid, ~300 for Montezuma).

Key Insights and Ideas

  • Pessimism Over Optimism: In online RL, exploration is usually driven by optimism under uncertainty. However, when dealing with learned reward models (like VLMs), the opposite is true: we need pessimism to guard against overconfident, misaligned reward signals.
  • False Positives vs. False Negatives: Missing a true reward (false negative) just leaves the landscape sparse, which intrinsic exploration can handle. But receiving a fake reward (false positive) actively misleads the agent into dead ends.
Figure 2: Schematic diagram of false positives in embedding space.
  • The Power of Synergy: BiMI works best when paired with an intrinsic reward model (like DEIR). BiMI provides sparse, highly reliable "anchor points" of task completion, while the intrinsic reward fills the gaps, driving efficient exploration between those anchors.

Key Results

  • Montezuma's Revenge: Integrating BiMI with the intrinsic reward model DEIR achieved a massive 88% performance boost over the baseline VLM reward model, significantly increasing the success rate of grabbing the first key.
  • Minigrid: The combination of ELLM- (a baseline VLM reward method) with BiMI and DEIR yielded a staggering 190% improvement over the baseline.
  • Ablation Studies: The binary reward (BI) mechanism alone accounted for a 36.5% performance increase across the board by successfully pruning out false positives.

Limitations and Future Work

  • Linear Instructions Only: The current framework assumes a strict, linear sequence of instructions. Real-world tasks often involve conditional ("if X, do Y") or ambiguous instructions, which this method does not yet handle.
  • The Crafter Exception: In the Crafter environment, the MI component actually decreased performance. This is because Crafter requires agents to repeatedly perform survival actions (like drinking water), and the MI regularization mistakenly penalized these necessary repetitions. Future work needs to distinguish between reward hacking and essential repetitive behaviors.
  • Non-Markovian Theory: While empirically successful on non-Markovian reward models (like video-based trajectory matching), the theoretical proofs are currently grounded in Markovian settings.

Summary updated

Abstract

While Vision-Language Models (VLMs) are increasingly used to generate reward signals for training embodied agents to follow instructions, our research reveals that agents guided by VLM rewards often underperform compared to those employing only intrinsic (exploration-driven) rewards, contradicting expectations set by recent work. We hypothesize that false positive rewards -- instances where unintended trajectories are incorrectly rewarded -- are more detrimental than false negatives. Our analysis confirms this hypothesis, revealing that the widely used cosine similarity metric is prone to false positive reward estimates. To address this, we introduce BiMI ({Bi}nary {M}utual {I}nformation), a novel reward function designed to mitigate noise. BiMI significantly enhances learning efficiency across diverse and challenging embodied navigation environments. Our findings offer a nuanced understanding of how different types of reward noise impact agent learning and highlight the importance of addressing multimodal reward signal noise when training embodied agents
BibTeX
@misc{huang2024dark,
  author = {Sukai Huang and Shu-Wei Liu and Nir Lipovetzky and Trevor Cohn},
  title = {{The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewards}},
  year = {2024},
  eprint = {2409.15922},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.15922},
}
Figure