Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models

Francisco Javier Chiyah Garcia, Alessandro Suglia, Arash EshghiEMNLP 2024 · DOI

Introduction

In the realm of human-robot collaboration, communication is rarely a linear, error-free process. When a robot misinterprets a command—such as identifying the wrong object to move—a human partner naturally provides a correction. In linguistics, this iterative process of clarifying a misunderstanding is known as a Third Position Repair (TPR). While humans navigate these repairs intuitively, current Vision and Language Models (VLMs) often struggle to update their internal state based on conversational feedback.

Figure 1: An illustration of the Third Position Repair (TPR) cycle in the Block World environment.

Most existing VLMs are trained as passive instruction followers, optimized for single-turn tasks. However, collaborative AI must be able to negotiate meaning and recover from miscommunication. To address this gap, researchers have introduced BLOCKWORLD-REPAIRS (BW-R), a benchmark specifically designed to evaluate how well models can process and learn from user corrections.

Methodology and Task Formulation

The BW-R framework evaluates VLMs in a simulated tabletop environment. The task requires models to process an image of a block world alongside a dialogue history to predict two sets of coordinates: the "Source Block" (the object to be moved) and the "Target Position" (the destination).

The study benchmarked several prominent models, including open-source architectures like Idefics2-8B and LLaVA-1.5-7B, as well as the proprietary GPT-4o. A core technical focus of the research was investigating how different training regimes affect a model's ability to handle repairs. The authors experimented with three Masking Token Loss strategies during fine-tuning:

  1. Default Loss: Loss is calculated across all tokens in the dialogue.
  2. User-Turn Loss: Loss is restricted to user inputs and the final target prediction.
  3. Completion-Only Loss: All tokens except the final correct response are masked. This strategy aims to prevent the model from "learning" its own incorrect intermediate steps.

The BLOCKWORLD-REPAIRS Dataset

The BW-R dataset expands upon the classic 2016 Block World dataset, providing a more complex foundation for multi-turn interaction. It consists of 795 collaborative dialogues, totaling 2,059 entries, collected via Amazon Mechanical Turk. Each entry follows a specific structure: an initial instruction, a simulated incorrect candidate response (the "misunderstanding"), and a subsequent user repair. To establish a high-quality baseline, the researchers also conducted an in-person study with 22 participants to determine how humans interpret these specific spatial repairs.

Key Results and Discussion

The experimental results highlight a significant performance gap between human capability and current AI models. While humans achieved 68% accuracy in initial source selection—rising to 75% after a repair—the best-performing fine-tuned model (LLaVA) reached only 54% accuracy on repairs.

Figure 6: Performance comparison of various models and training regimes on the BW-R benchmark.

A critical finding was the "Negative Learning" Trap. Standard fine-tuning on full dialogue histories can actually degrade performance. If a model is trained on its own mistakes, it may inadvertently strengthen the association between the original (misunderstood) instruction and the wrong coordinates. The "Completion-only" loss emerged as the most effective solution, significantly boosting Idefics2's repair accuracy from 26% to 47%.

Furthermore, the models demonstrated a clear hierarchy of difficulty regarding spatial reasoning. While proficient with basic terms like "left" or "above," models struggled with abstract organizational concepts such as "rows" or "columns." Additionally, predicting the "Target Position" proved much harder than identifying a "Source Block," suggesting that current VLM pre-training lacks sufficient reasoning about 3D spatial voids.

Conclusion and Future Directions

The BW-R benchmark reveals that while VLMs show promise, they are not yet robust collaborative partners. Future work must move beyond static annotation tasks toward "closed-loop" evaluations where model actions dynamically alter the environment. Additionally, the authors suggest that incorporating object-centric visual encoders and moving toward direct robotic control API generation will be essential steps in creating AI that can truly "negotiate" tasks with human users in real-time.

Summary updated

BibTeX
@inproceedings{garcia2024repairs,
  author = {Francisco Javier Chiyah Garcia and Alessandro Suglia and Arash Eshghi},
  title = {{Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.643},
}
Figure