DiNeR: a Large Realistic Dataset for Evaluating Compositional Generalization

Chengang Hu, Xiao Liu, Yansong Feng

Abstract

Most of the existing compositional generalization datasets are synthetically-generated, resulting in a lack of natural language variation. While there have been recent attempts to introduce non-synthetic datasets for compositional generalization, they suffer from either limited data scale or a lack of diversity in the forms of combinations. To better investigate compositional generalization with more linguistic phenomena and compositional diversity, we propose the DIsh NamE Recognition (DiNeR) task and create a large realistic Chinese dataset. Given a recipe instruction, models are required to recognize the dish name composed of diverse combinations of food, actions, and flavors. Our dataset consists of 3,811 dishes and 228,114 recipes, and involves plenty of linguistic phenomena such as anaphora, omission and ambiguity. We provide two strong baselines based on T5 and large language models (LLMs). This work contributes a challenging task, baseline methods to tackle the task, and insights into compositional generalization in the context of dish name recognition.

Introduction

Compositional generalization—the ability of a system to understand and produce a potentially infinite number of novel combinations of known components—is a hallmark of human intelligence. For instance, a person who understands "braised pork" and "beef brisket" can easily infer the concept of "braised beef brisket," even if they have never encountered that specific dish.

The DiNeR task: inferring a new dish name from known components in other recipes.

Current benchmarks for compositional generalization often rely on synthetic "toy" datasets, which lack the linguistic nuances found in the real world. Real-world cooking instructions are replete with complex phenomena such as anaphora (referring to "brisket" later as "meat pieces"), omission (e.g., "stir-fry well" without explicitly stating the object), and lexical ambiguity. To bridge this gap, we introduce the DIsh NamE Recognition (DiNeR) task and dataset.

The DiNeR Dataset

The DiNeR dataset is constructed from the XIACHUFANG recipe platform, comprising 3,811 unique dishes and 228,114 recipes. Unlike previous datasets, DiNeR is characterized by its massive scale and the diversity of its generalization forms, offering a more realistic testbed for AI models.

Comparison of DiNeR with existing compositional generalization datasets.

To rigorously test generalization, we employ the Target Maximum Compound Divergence (TMCD) split. This ensures that the test set contains combinations of ingredients, actions, and flavors that are never seen together in the training set, despite the individual components being familiar. A pilot study revealed that linguistic challenges like anaphora and omissions occur in nearly every sample, making this a formidable benchmark.

Methodology and Baselines

We evaluate several modeling strategies on the DiNeR task:

  1. Continual Pretraining: Enhancing a T5 model with culinary domain knowledge by pretraining on a large recipe corpus.
  2. Compositional Prompting (CP-FT): A multi-stage approach where the model first identifies sub-components (Food, Action, Flavor) before predicting the final dish name.
  3. LLM In-Context Learning: Evaluating GPT-3.5 using selective demonstrations to guide its reasoning through similar examples.

Key Findings and Insights

Our experiments reveal several critical insights into the state of compositional generalization:

  • The OOD Performance Gap: There is a significant drop in performance when moving from in-distribution (random split) to out-of-distribution (TMCD split) test sets. A standard T5 model's F1 score plummets from 82.4 to 50.8 when faced with novel combinations.
  • The Inverse-Scaling Phenomenon: Surprisingly, under heavy distribution shifts, increasing the amount of training data can lead to a decrease in OOD performance. This suggests that intensive fine-tuning may cause the model to overfit to specific distributional patterns, reducing its flexibility for novel synthesis.
  • Effectiveness of Sub-tasks: Explicitly identifying components through Compositional Prompting significantly improves performance. By forcing the model to recognize "this is beef" and "this is a braising action," it becomes better equipped to synthesize the name "Braised Beef."
  • LLM Limitations: Despite its general-purpose capabilities, GPT-3.5 (52.9 F1) was outperformed by our specialized T5 baseline (59.2 F1) on the OOD task, highlighting the difficulty of mastering specific cultural naming conventions in a zero-shot or few-shot manner.

Conclusion

DiNeR provides a challenging and realistic benchmark for testing the compositional limits of modern NLP models. The observed "inverse-scaling" effect and the persistent gap between human-like reasoning and model performance suggest that new architectural or training paradigms are needed to achieve true linguistic compositionality in real-world domains.

Summary updated

Abstract

Most of the existing compositional generalization datasets are synthetically-generated, resulting in a lack of natural language variation. While there have been recent attempts to introduce non-synthetic datasets for compositional generalization, they suffer from either limited data scale or a lack of diversity in the forms of combinations. To better investigate compositional generalization with more linguistic phenomena and compositional diversity, we propose the DIsh NamE Recognition (DiNeR) task and create a large realistic Chinese dataset. Given a recipe instruction, models are required to recognize the dish name composed of diverse combinations of food, actions, and flavors. Our dataset consists of 3,811 dishes and 228,114 recipes, and involves plenty of linguistic phenomena such as anaphora, omission and ambiguity. We provide two strong baselines based on T5 and large language models (LLMs). This work contributes a challenging task, baseline methods to tackle the task, and insights into compositional generalization in the context of dish name recognition. Code and data are available at https://github.com/Jumpy-pku/DiNeR.
BibTeX
@inproceedings{hu2023diner,
  author = {Chengang Hu and Xiao Liu and Yansong Feng},
  title = {{DiNeR: a Large Realistic Dataset for Evaluating Compositional Generalization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.924},
}
Figure