Abstract
While Vision-Language Models (VLMs) are increasingly used to generate reward signals for training embodied agents to follow instructions, our research reveals that agents guided by VLM rewards often underperform compared to those employing only intrinsic (exploration-driven) rewards, contradicting expectations set by recent work. We hypothesize that false positive rewards -- instances where unintended trajectories are incorrectly rewarded -- are more detrimental than false negatives. Our analysis confirms this hypothesis, revealing that the widely used cosine similarity metric is prone to false positive reward estimates. To address this, we introduce BiMI (Binary Mutual Information), a novel reward function designed to mitigate noise. BiMI significantly enhances learning efficiency across diverse and challenging embodied navigation environments. Our findings offer a nuanced understanding of how different types of reward noise impact agent learning and highlight the importance of addressing multimodal reward signal noise when training embodied agents.
Introduction
Vision-Language Models (VLMs) are increasingly utilized to bridge the gap between natural language instructions and robotic actions by providing "rich" auxiliary reward signals. However, recent research suggests a paradox: agents guided by these sophisticated models often underperform compared to those relying on simple, intrinsic exploration rewards. This paper investigates this phenomenon, identifying reward noise—specifically false positives—as the primary deterrent to effective learning in embodied agents.
The Challenge of Reward Noise
In sparse-reward environments, VLMs are intended to provide dense feedback by measuring the similarity between an agent's visual trajectory and a textual instruction. The standard metric for this, cosine similarity, suffers from "compositional insensitivity." It frequently assigns high rewards to trajectories that are lexically similar but semantically incorrect, such as performing actions in the wrong order or interacting with the wrong object (e.g., a red pill instead of a blue one).
The authors demonstrate that these false positives (overestimating the value of an action) are significantly more damaging to Reinforcement Learning (RL) convergence than false negatives (underestimating value). When an agent is incorrectly rewarded for a failure, it reinforces detrimental behaviors, leading to local minima or catastrophic failures.
Methodology: The BiMI Reward Function
To mitigate the impact of noisy VLM signals, the authors propose BiMI (Binary Mutual Information). This reward function shifts the focus from continuous similarity scores to high-confidence binary events.
BiMI consists of two primary components:
- Binary Signal (BI): Utilizing conformal prediction, the model sets a high-confidence threshold. The agent receives a discrete reward (+1) only when the VLM is statistically certain the task has been completed. This "pessimistic" approach prioritizes accuracy over density, effectively pruning false positives.
- Mutual Information (MI): This term acts as a regularizer, penalizing rewards for signals that occur too frequently. This prevents "reward hacking," where an agent might repeat a simple, irrelevant action that the VLM happens to score highly.
Evaluation and Results
The effectiveness of BiMI was tested across three complex, long-horizon environments: Montezuma’s Revenge, Crafter, and Minigrid. These environments were chosen specifically because they expose how reward noise accumulates over time.
The results indicate that "cleaning" the reward signal is more impactful than increasing model size:
- Montezuma’s Revenge: BiMI achieved an 88% improvement over baseline VLM reward models when paired with intrinsic motivation.
- Minigrid: In specific configurations, performance gains reached up to +364%.
- Ablation: The binary thresholding component alone contributed to a ~36% performance boost, validating the hypothesis that filtering noise is critical.
Conclusion and Future Work
While BiMI significantly improves performance in many scenarios, it faces challenges in environments like Crafter, where necessary repetitive actions (e.g., drinking water) can be incorrectly penalized by the MI term. Furthermore, while this study focused on encoder-based models like CLIP, the authors anticipate that the "dark side" of rewards—hallucinated feedback—will likely persist or even intensify with newer generative VLMs. Future research will focus on handling complex conditional logic and ambiguous instructions within the BiMI framework.
Summary updated