Abstract
Compositional generalization—the ability to understand novel combinations of known components—is a hallmark of human intelligence. However, existing benchmarks often rely on synthetic data that lacks the linguistic complexity of real-world language. In this work, researchers from Peking University introduce DINER, a massive, realistic Chinese recipe dataset designed to test these limits. DINER incorporates natural language nuances such as anaphora and omissions, revealing a surprising "inverse-scaling" effect: in high-shift scenarios, more training data can actually hinder a model's ability to generalize.
Introduction
The capacity for compositional generalization allows humans to easily conceptualize "braised beef brisket" if they already understand "braised pork" and "marinated beef brisket." While modern AI has made significant progress, most evaluation datasets are synthetic and structured, lacking the "messiness" of natural language. Real-world language is characterized by anaphora (referring to "beef brisket" as "the meat pieces"), omissions (saying "stir fry well" without an explicit object), and contextual ambiguity.
The DINER Dataset
To address these gaps, the authors present the DIsh NamE Recognition (DINER) task. Derived from the XIACHUFANG recipe corpus, the dataset contains 228,114 recipes covering 3,811 unique dishes. Unlike previous datasets, DINER is significantly larger and includes four distinct forms of generalization.
The researchers used Target Maximum Compound Divergence (TMCD) to split the data, ensuring the test set contains combinations of ingredients and techniques never seen together during training. This creates a rigorous Out-of-Distribution (OOD) challenge.
Methodology
The authors evaluated several strategies to improve generalization:
- Compositional Prompting Fine-Tuning (CP-FT): An auxiliary model identifies individual components (food, action, flavor), which are then used as prompts for the main model to summarize the final dish name.
- Continual Pretraining: A T5 model is pretrained on a large recipe corpus to gain domain-specific culinary knowledge before task-specific fine-tuning.
- LLM Baselines: GPT-3.5 was tested using selective demonstrations to provide context for the task.
Key Results and Insights
The study revealed a significant "generalization gap." A standard T5 model achieved an 82.4 F1 score on in-distribution data but dropped to 50.8 F1 on the OOD split. However, the most provocative finding was the inverse-scaling phenomenon.
As shown in the experimental results, increasing the training data size actually decreased performance on high-shift OOD sets. The authors hypothesize that extensive fine-tuning on a specific distribution "distorts" pretrained features, making the model too rigid for novel combinations.
Furthermore, the complexity of natural language proved challenging. Models struggled with anaphora and omissions, which require true abstraction rather than simple keyword extraction. Flavor components were the most difficult to predict, likely due to their lower frequency in the training data.
Conclusion
DINER provides a challenging new benchmark for compositional generalization. The results suggest that simply adding more data is not a panacea for OOD performance. Future research must focus on preventing "feature distortion" during fine-tuning to ensure models retain the flexibility needed to handle the infinite combinations of the real world.
Summary updated