${lil}$Gym: Natural Language Visual Reasoning with Reinforcement Learning

Anne Wu, Kianté Brantley, Noriyuki Kojima, Yoav Artzi

TL;DR

This paper introduces lilGym, a new reinforcement learning benchmark designed to test agents' ability to perform visual reasoning based on natural language instructions. It leverages the rich, compositional language of the NLVR dataset and provides executable programs for precise reward calculation, creating a challenging environment for current RL methods.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents are increasingly being trained to understand and act upon natural language instructions. However, creating realistic benchmarks that capture the complexity of natural language and its grounding in visual environments has been a significant challenge. Existing approaches often simplify language or rely on heuristic reward functions, which can lead to agents learning superficial correlations rather than true reasoning. This paper addresses this gap by proposing lilGym, a benchmark that uses diverse, human-written language and provides exact reward computation, pushing the boundaries of language-conditioned RL.

Methods

lilGym is built upon the Natural Language for Visual Reasoning (NLVR) corpus. The core innovation is the creation of executable Python programs for each NLVR statement. These programs act as ground truth evaluators, allowing for precise reward calculation in any given visual state. This enables the formulation of Contextual Markov Decision Processes (CMDPs), where the context is the natural language statement and a target boolean value. The environments are categorized into four configurations: TOWER-SCRATCH, TOWER-FLIPIT, SCATTER-SCRATCH, and SCATTER-FLIPIT, differing in object placement rules and initial conditions. The paper evaluates standard RL algorithms, specifically Proximal Policy Optimization (PPO), with two visuo-linguistic models: C+BERT and ViLT. They also introduce a variant, PPO+SF (PPO with stop forcing), to aid exploration during training.

Datasets

The primary dataset used is the Natural Language for Visual Reasoning (NLVR) corpus. lilGym leverages the existing NLVR data and adds new annotations: executable Python programs for each statement. This allows for the creation of thousands of distinct Markov Decision Processes (MDPs) across four configurations (TOWER-SCRATCH, TOWER-FLIPIT, SCATTER-SCRATCH, SCATTER-FLIPIT) for training, development, and testing.

Key Insights and Ideas

  • Exact Reward Computation is Crucial: The paper highlights the importance of precise reward signals derived from executable semantic parsers. This moves beyond approximate rewards and forces agents to truly understand the language.
  • Compositional Language is Hard: The NLVR corpus, with its highly compositional human-written language, presents a significant challenge for RL agents, requiring them to reason about relationships between objects, properties, and spatial arrangements.
  • Exploration is a Major Hurdle: The SCATTER configurations, with their large action spaces, demonstrate a hard exploration problem for RL agents, underscoring the need for more sophisticated exploration strategies.
  • Model Architecture Matters: ViLT, a pre-trained multi-modal Transformer, generally outperforms C+BERT, suggesting the benefits of joint visuo-linguistic pre-training for this task.

Key Results

The paper reports accuracies for different model-algorithm pairs across the four lilGym configurations.

  • General Performance: Existing methods achieve non-trivial performance, but accuracies are far from perfect, especially in the more complex SCATTER configurations.
  • PPO+SF Improvement: The PPO+SF variant shows improved performance, particularly on SCATTER CMDPs, by aiding exploration during training. For instance, on SCATTER-FLIPIT, PPO+SF improves performance by 23.86% compared to PPO.
  • Model Comparison: ViLT generally outperforms C+BERT, with accuracies reaching up to 84.05% in TOWER-SCRATCH (PPO+SF) and 70.97% in SCATTER-FLIPIT (PPO+SF), while C+BERT achieves 80.98% and 8.46% respectively.
  • Mistake Analysis: For PPO-trained models on SCATTER-SCRATCH, a significant portion of mistakes (76%) are due to invalid actions (e.g., trying to place an object that doesn't fit, acting on separators) and 24% due to early termination.

Limitations and Future Work

  • Limitations: The paper acknowledges that while existing methods achieve non-trivial performance, lilGym presents a challenging open problem, indicating room for significant improvement. The analysis of mistakes points to specific areas where agents struggle, such as handling invalid actions and effective exploration.
  • Future Work: The authors suggest that future work should report both development and held-out test results for easier comparison. They also note that the current work does not manipulate the "box element permutations" property of the NLVR data, which could be explored in future research. The development of more robust semantic parsers is also mentioned as a way to automate annotation processes.

Summary updated

Abstract

We present ${lil}$Gym, a new benchmark for language-conditioned reinforcement learning in visual environments. ${lil}$Gym is based on 2,661 highly-compositional human-written natural language statements grounded in an interactive visual environment. We annotate all statements with executable Python programs representing their meaning to enable exact reward computation in every possible world state. Each statement is paired with multiple start states and reward functions to form thousands of distinct Markov Decision Processes of varying difficulty. We experiment with ${lil}$Gym with different models and learning regimes. Our results and analysis show that while existing methods are able to achieve non-trivial performance, ${lil}$Gym forms a challenging open problem. ${lil}$Gym is available at https://lil.nlp.cornell.edu/lilgym/.
BibTeX
@inproceedings{wu2023lil,
  author = {Anne Wu and Kianté Brantley and Noriyuki Kojima and Yoav Artzi},
  title = {{$\{lil\}$Gym: Natural Language Visual Reasoning with Reinforcement Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.512},
}
Figure