Abstract
lilGym is a new reinforcement learning (RL) benchmark designed to evaluate agents on their ability to manipulate visual environments to satisfy complex, human-written natural language statements. By pairing thousands of human sentences with executable Python programs for exact reward computation, lilGym provides a rigorous framework for studying grounded language understanding. Our results reveal a significant performance gap between state-of-the-art models and human-level reasoning, particularly in tasks requiring complex spatial logic and high-dimensional exploration.
Introduction
Training agents to connect linguistic concepts to visual objects and actions—known as grounded language learning—is a fundamental goal in AI. However, current RL benchmarks often rely on synthetic, templated language or simplify tasks to reaching a single goal state. These simplifications fail to capture the compositional complexity and abstractive nature of human communication, where a single statement like "There are two boxes which has the same number of objects and three kinds of colors" can be satisfied by a vast array of configurations.
To address these limitations, we introduce lilGym, a benchmark that tasks agents with modifying a 2D environment to make a natural language statement true or false.
The lilGym Benchmark
lilGym is built upon the Natural Language for Visual Reasoning (NLVR) corpus, featuring 2,661 human-written statements. The benchmark is categorized into four distinct settings based on object arrangement and initial conditions:
- TOWER-SCRATCH: Objects are stacked in fixed columns, starting from an empty board.
- TOWER-FLIPIT: Objects are stacked, starting from an existing configuration that must be modified.
- SCATTER-SCRATCH: Objects are placed freely on a 2D plane, starting from an empty board.
- SCATTER-FLIPIT: Objects are placed freely, starting from an existing configuration.
Methods: Executable Rewards and Architectures
The primary challenge in using natural language for RL rewards is the ambiguity of success. lilGym solves this by treating language grounding as a programming task. Expert programmers translated each human statement into an executable Python program that serves as a "judge," providing an exact boolean reward based on the environment state.
Agents can perform ADD or REMOVE actions and must learn to emit a STOP action when the goal is met. We introduce Stop Forcing (SF), a training technique that masks all actions except STOP once a successful state is reached, significantly improving the agent's ability to terminate trajectories correctly.
We evaluated several architectures, including:
- C3/C10+BERT: Combining CNN-based visual encoders with BERT for text processing.
- ViLT (Vision-and-Language Transformer): A model that jointly processes image patches and text tokens using a unified transformer architecture.
Results and Discussion
Our experiments show that joint reasoning is crucial for success. ViLT consistently outperformed separate-stream models, achieving 81.48% accuracy in the TOWER-SCRATCH setting. However, performance drops sharply in SCATTER environments, where the action space exceeds one million possible configurations. In the most challenging SCATTER-FLIPIT task, the best model achieved only 30.06% accuracy.
Key insights include:
- Logic as a Bottleneck: Agents struggle with complex logical constraints, such as "hard cardinality" (e.g., "exactly four") or precise spatial relations.
- Exploration Challenges: The high-dimensional action space in
SCATTERtasks leads to frequent invalid actions, indicating a need for more sophisticated exploration strategies. - Human-AI Gap: While human experts achieve 100% accuracy with minimal steps, RL agents often "fumble," taking inefficient trajectories that suggest a lack of deep logical understanding.
Conclusion
lilGym provides a challenging new frontier for grounded language learning. By moving beyond synthetic templates to human logic and exact rewards, it highlights the current limitations of RL agents in compositional reasoning and high-dimensional manipulation. Future work will explore program synthesis and more robust exploration methods to bridge the gap toward human-level performance.
Summary updated