Abstract
Abstraction is a core tenet of human cognition and communication. When composing natural language instructions, humans naturally evoke abstraction to convey complex procedures in an efficient and concise way. Yet, interpreting and grounding abstraction expressed in NL has not yet been systematically studied in NLP, with no accepted benchmarks specifically eliciting abstraction in NL. In this work, we set the foundation for a systematic study of processing and grounding abstraction in NLP. First, we deliver a novel abstraction elicitation method and present Hexagons, a 2D instruction-following game. Using Hexagons we collected over 4k naturally-occurring visually-grounded instructions rich with diverse types of abstractions. From these data, we derive an instruction-to-execution task and assess different types of neural models. Our results show that contemporary models and modeling practices are substantially inferior to human performance, and that models' performance is inversely correlated with the level of abstraction, showing less satisfying performance on higher levels of abstraction. These findings are consistent across models and setups, confirming that abstraction is a challenging phenomenon deserving further attention and study in NLP/AI research.
Introduction
Human communication relies heavily on abstraction. We use high-level concepts like loops, conditions, and object definitions to convey complex procedures efficiently. For instance, instead of describing a pattern tile-by-tile, a human might say, "Repeat this pattern across the board." While AI models have become proficient at following literal, low-level instructions, their ability to ground formal abstract structures expressed in informal natural language remains largely unexplored. This research introduces HEXAGONS, a novel benchmark designed to systematically study and evaluate how neural models handle abstraction in a visually-grounded environment.
The HEXAGONS Benchmark
To elicit natural abstract language, the researchers developed a 2D instruction-following game. In HEXAGONS, an "Instructor" describes a target image—often featuring fractals, symmetries, or repeating shapes—to an "Executor" who recreates it on a 180-tile hexagonal grid. The complexity of the visual stimuli encourages participants to use abstract language to avoid the tedium of literal descriptions.
The resulting HEXAGONS dataset contains 4,177 drawing steps across 620 unique procedures. Each instruction is manually annotated with one of four levels of abstraction, ranging from Level 0 (literal coordinates) to Level 3 (complex iterations and conditions).
Experimental Results and the "Abstraction Tax"
The study evaluated two primary neural architectures: a classification-based model (DeBERTa) and a generation-based model (T5). The task was framed as instruction-to-execution, where models predict the final state of the board based on the text.
A significant finding is the "Abstraction Tax": a clear inverse correlation between the level of abstraction and model performance. While models perform reasonably well on literal (Level 0) instructions, their accuracy drops precipitously as abstraction increases. On Level 3 instructions, the best model achieved a Mean F1 score of only 3.59, compared to a human baseline of 83.52.
Key insights include:
- Contextual Dependency: Models benefit significantly from the "Full History" of instructions, suggesting that understanding the logical progression of a pattern is crucial.
- Error Propagation: In abstract tasks like loops, a single initial error cascades, leading to total failure in exact match metrics.
- Data Scaling Limits: Simply increasing training data size does not bridge the gap for high-level abstraction, indicating a need for architectural innovations.
Conclusion
The HEXAGONS benchmark reveals a stark gap between human and machine performance in grounding abstraction. Current models struggle with the logical coordination required by abstract natural language. Future research should focus on architectures capable of explicit abstraction detection, recursive logic, and error recovery to move closer to human-like instruction following.
Summary updated