Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking

Ronen Tamari, Kyle Richardson, Aviad Sar-Shalom, Noam Kahlon, Nelson F. Liu, Reut Tsarfaty, Dafna Shahaf

Abstract

Dyna-bAbI is a dynamic benchmarking framework designed to stress-test story understanding in natural language models. By transitioning from static datasets to a controllable task generator, the researchers demonstrate that models achieving near-perfect scores on the original bAbI benchmark often fail when handling novel combinations of simple concepts. This reveals a significant gap in "compositional generalization," where models struggle to integrate separately learned reasoning skills.

Introduction: The Problem of Benchmark Saturation

In Natural Language Understanding (NLU), the bAbI benchmark has been a standard for testing reasoning over simple narratives. However, modern models have reached a state of "saturation," consistently achieving over 99% accuracy. This success is often illusory, driven by "annotation artifacts"—subtle biases that allow models to bypass actual logic. Crucially, static benchmarks fail to test compositional generalization: the ability to combine distinct concepts (e.g., object tracking and co-reference) in novel, unseen contexts.

Comparison of static vs. dynamic benchmarking approaches in Dyna-bAbI.

Methods: Dynamic Task Generation

To address these limitations, the researchers developed Dyna-bAbI, a Python-based task generator. Unlike hard-coded scripts, Dyna-bAbI provides "knobs" to manipulate task difficulty and structure, allowing for fine-grained control over passage length and concept sets.

The framework introduces several training and testing regimes:

  • Concat: Standard concatenation of original tasks.
  • Inject: Supplementing stories with additional questions for higher supervision.
  • Diverse: Using rejection sampling to create complex "supporting fact" structures.
  • Mix: The primary challenge set, featuring out-of-distribution (OOD) combinations of concepts never seen together during training.

Evaluations were conducted across specialized bAbI architectures (e.g., EntNet), general-purpose QA methods (BiDAF), and large pre-trained transformers (RoBERTa, T5).

Key Insights and Results

The study reveals that mastery of individual components does not guarantee mastery of the whole. Models acing separate tasks often fail when those elements are combined, suggesting they learn specific "templates" rather than general reasoning rules.

The Performance Cliff

On the "7-task OOD" split, specialized models like EntNet saw accuracy plummet from ~99% to roughly 22-26%. Even state-of-the-art models like T5 experienced a significant drop to 62.6%. While pre-training provides a better baseline, it does not inherently grant robust logic; T5 still suffered accuracy losses of 30-50% on complex tasks.

Concurrence with Real-World Data

A significant finding is that Dyna-bAbI performance is a better predictor of real-world efficacy than the original benchmark. The researchers found that Dyna-bAbI has a much higher "concurrence" with datasets like SQuAD, meaning it more accurately reflects a model's ability to handle human-language tasks.

Correlation between model performance on bAbI/Dyna-bAbI and SQuAD EM scores.

Knowledge Inconsistency

Models also displayed "brittleness" to phrasing. For instance, a model might correctly identify a character's location in an open-ended question but fail a corresponding Yes/No question. This indicates that the model's internal "situation model" is tied to specific question formats rather than the underlying facts.

Conclusion and Future Work

While Dyna-bAbI uses synthetic language, it effectively exposes logical flaws in current NLU models. Future research will focus on integrating more complex reasoning tasks (e.g., spatial or motivational) and developing training methods that encourage the formation of robust, question-independent "situation models."

Summary updated

BibTeX
@inproceedings{tamari2022dyna,
  author = {Ronen Tamari and Kyle Richardson and Aviad Sar-Shalom and Noam Kahlon and Nelson F. Liu and Reut Tsarfaty and Dafna Shahaf},
  title = {{Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking}},
  booktitle = {SEM @ NAACL},
  year = {2022},
  doi = {10.18653/v1/2022.starsem-1.9},
}
Figure