TL;DR
This paper introduces lilGym, a new reinforcement learning benchmark designed to test agents' ability to perform visual reasoning based on natural language instructions. It leverages the rich, compositional language of the NLVR dataset and provides executable programs for precise reward calculation, creating a challenging environment for current RL methods.
What Problem Does This Paper Address?
Reinforcement learning (RL) agents are increasingly being trained to understand and act upon natural language instructions. However, creating realistic benchmarks that capture the complexity of natural language and its grounding in visual environments has been a significant challenge. Existing approaches often simplify language or rely on heuristic reward functions, which can lead to agents learning superficial correlations rather than true reasoning. This paper addresses this gap by proposing lilGym, a benchmark that uses diverse, human-written language and provides exact reward computation, pushing the boundaries of language-conditioned RL.
Methods
lilGym is built upon the Natural Language for Visual Reasoning (NLVR) corpus. The core innovation is the creation of executable Python programs for each NLVR statement. These programs act as ground truth evaluators, allowing for precise reward calculation in any given visual state. This enables the formulation of Contextual Markov Decision Processes (CMDPs), where the context is the natural language statement and a target boolean value. The environments are categorized into four configurations: TOWER-SCRATCH, TOWER-FLIPIT, SCATTER-SCRATCH, and SCATTER-FLIPIT, differing in object placement rules and initial conditions. The paper evaluates standard RL algorithms, specifically Proximal Policy Optimization (PPO), with two visuo-linguistic models: C+BERT and ViLT. They also introduce a variant, PPO+SF (PPO with stop forcing), to aid exploration during training.
Datasets
The primary dataset used is the Natural Language for Visual Reasoning (NLVR) corpus. lilGym leverages the existing NLVR data and adds new annotations: executable Python programs for each statement. This allows for the creation of thousands of distinct Markov Decision Processes (MDPs) across four configurations (TOWER-SCRATCH, TOWER-FLIPIT, SCATTER-SCRATCH, SCATTER-FLIPIT) for training, development, and testing.
Key Insights and Ideas
- Exact Reward Computation is Crucial: The paper highlights the importance of precise reward signals derived from executable semantic parsers. This moves beyond approximate rewards and forces agents to truly understand the language.
- Compositional Language is Hard: The NLVR corpus, with its highly compositional human-written language, presents a significant challenge for RL agents, requiring them to reason about relationships between objects, properties, and spatial arrangements.
- Exploration is a Major Hurdle: The SCATTER configurations, with their large action spaces, demonstrate a hard exploration problem for RL agents, underscoring the need for more sophisticated exploration strategies.
- Model Architecture Matters: ViLT, a pre-trained multi-modal Transformer, generally outperforms C+BERT, suggesting the benefits of joint visuo-linguistic pre-training for this task.
Key Results
The paper reports accuracies for different model-algorithm pairs across the four lilGym configurations.
- General Performance: Existing methods achieve non-trivial performance, but accuracies are far from perfect, especially in the more complex SCATTER configurations.
- PPO+SF Improvement: The PPO+SF variant shows improved performance, particularly on SCATTER CMDPs, by aiding exploration during training. For instance, on SCATTER-FLIPIT, PPO+SF improves performance by 23.86% compared to PPO.
- Model Comparison: ViLT generally outperforms C+BERT, with accuracies reaching up to 84.05% in TOWER-SCRATCH (PPO+SF) and 70.97% in SCATTER-FLIPIT (PPO+SF), while C+BERT achieves 80.98% and 8.46% respectively.
- Mistake Analysis: For PPO-trained models on SCATTER-SCRATCH, a significant portion of mistakes (76%) are due to invalid actions (e.g., trying to place an object that doesn't fit, acting on separators) and 24% due to early termination.
Limitations and Future Work
- Limitations: The paper acknowledges that while existing methods achieve non-trivial performance,
lilGympresents a challenging open problem, indicating room for significant improvement. The analysis of mistakes points to specific areas where agents struggle, such as handling invalid actions and effective exploration. - Future Work: The authors suggest that future work should report both development and held-out test results for easier comparison. They also note that the current work does not manipulate the "box element permutations" property of the NLVR data, which could be explored in future research. The development of more robust semantic parsers is also mentioned as a way to automate annotation processes.
Summary updated