DiNeR: A Large Realistic Dataset for Evaluating Compositional Generalization

Chengang Hu, Xiao Liu, Yansong Feng

Abstract

Compositional generalization—the ability to understand novel combinations of known components—is a hallmark of human intelligence. However, existing benchmarks often rely on synthetic data that lacks the linguistic complexity of real-world language. In this work, researchers from Peking University introduce DINER, a massive, realistic Chinese recipe dataset designed to test these limits. DINER incorporates natural language nuances such as anaphora and omissions, revealing a surprising "inverse-scaling" effect: in high-shift scenarios, more training data can actually hinder a model's ability to generalize.

Introduction

The capacity for compositional generalization allows humans to easily conceptualize "braised beef brisket" if they already understand "braised pork" and "marinated beef brisket." While modern AI has made significant progress, most evaluation datasets are synthetic and structured, lacking the "messiness" of natural language. Real-world language is characterized by anaphora (referring to "beef brisket" as "the meat pieces"), omissions (saying "stir fry well" without an explicit object), and contextual ambiguity.

The DINER Dataset

To address these gaps, the authors present the DIsh NamE Recognition (DINER) task. Derived from the XIACHUFANG recipe corpus, the dataset contains 228,114 recipes covering 3,811 unique dishes. Unlike previous datasets, DINER is significantly larger and includes four distinct forms of generalization.

The DINER task: predicting a new dish name by composing elements (actions, ingredients, flavors) from known cooking instructions.

The researchers used Target Maximum Compound Divergence (TMCD) to split the data, ensuring the test set contains combinations of ingredients and techniques never seen together during training. This creates a rigorous Out-of-Distribution (OOD) challenge.

Methodology

The authors evaluated several strategies to improve generalization:

  1. Compositional Prompting Fine-Tuning (CP-FT): An auxiliary model identifies individual components (food, action, flavor), which are then used as prompts for the main model to summarize the final dish name.
  2. Continual Pretraining: A T5 model is pretrained on a large recipe corpus to gain domain-specific culinary knowledge before task-specific fine-tuning.
  3. LLM Baselines: GPT-3.5 was tested using selective demonstrations to provide context for the task.

Key Results and Insights

The study revealed a significant "generalization gap." A standard T5 model achieved an 82.4 F1 score on in-distribution data but dropped to 50.8 F1 on the OOD split. However, the most provocative finding was the inverse-scaling phenomenon.

Inverse-scaling effect: as the training data size increases (x-axis), model performance (y-axis) decreases for high-divergence OOD test sets.

As shown in the experimental results, increasing the training data size actually decreased performance on high-shift OOD sets. The authors hypothesize that extensive fine-tuning on a specific distribution "distorts" pretrained features, making the model too rigid for novel combinations.

Furthermore, the complexity of natural language proved challenging. Models struggled with anaphora and omissions, which require true abstraction rather than simple keyword extraction. Flavor components were the most difficult to predict, likely due to their lower frequency in the training data.

Conclusion

DINER provides a challenging new benchmark for compositional generalization. The results suggest that simply adding more data is not a panacea for OOD performance. Future research must focus on preventing "feature distortion" during fine-tuning to ensure models retain the flexibility needed to handle the infinite combinations of the real world.

Summary updated

Abstract

Most of the existing compositional generalization datasets are synthetically-generated, resulting in a lack of natural language variation. While there have been recent attempts to introduce non-synthetic datasets for compositional generalization, they suffer from either limited data scale or a lack of diversity in the forms of combinations. To better investigate compositional generalization with more linguistic phenomena and compositional diversity, we propose the DIsh NamE Recognition (DiNeR) task and create a large realistic Chinese dataset. Given a recipe instruction, models are required to recognize the dish name composed of diverse combinations of food, actions, and flavors. Our dataset consists of 3,811 dishes and 228,114 recipes, and involves plenty of linguistic phenomena such as anaphora, omission and ambiguity. We provide two strong baselines based on T5 and large language models (LLMs). This work contributes a challenging task, baseline methods to tackle the task, and insights into compositional generalization in the context of dish name recognition. Code and data are available at https://github.com/Jumpy-pku/DiNeR.
BibTeX
@inproceedings{hu2023diner,
  author = {Chengang Hu and Xiao Liu and Yansong Feng},
  title = {{DiNeR: A Large Realistic Dataset for Evaluating Compositional Generalization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.924},
}
Figure