Abstract
Large Language Models (LLMs) excel at linguistic processing but often lack a grounded understanding of the physical world. In procedural contexts like cooking, tracking the evolving state of ingredients is crucial for true comprehension. This paper presents a novel, highly clean Japanese recipe dataset annotated with ingredient state changes. We introduce "state probing" tasks to evaluate how models track physical transformations. Our findings indicate that while standard LLMs struggle with implicit state changes, fine-tuning on structured world-state data—particularly with multimodal models—significantly enhances their physical reasoning capabilities.
Introduction
Large Language Models (LLMs) are remarkably proficient at text generation, yet they remain largely "blind" to the physical world. When reading a recipe, an LLM processes the text linguistically without necessarily building a mental model of the physical environment. Procedural texts often leave physical details implicit, assuming the reader understands the current state of ingredients—for example, that eggs mentioned in a later step were already boiled and peeled previously. This "state tracking" problem is a significant hurdle for models attempting to understand complex procedures.
Methodology
The researchers employ a technique called State Probing, which treats the "world state" as a set of natural language descriptions of every item in the kitchen at a specific time.
Graph-Based Annotation
Recipes are converted into Directed Acyclic Graphs (DAGs), where nodes represent ingredient states and edges represent transformations. This structured approach allows for precise tracking of how ingredients evolve, split, or combine.
State Probing Tasks
Three novel evaluation tasks were defined to measure model performance:
- Step Reference Resolution: Predicting which previous step a masked reference refers to (e.g., "Combine the mixture from [Step X]").
- Ingredient Usage Prediction: A binary task determining if a specific initial ingredient has been used or remains in its original state.
- Ingredient Tracing: Identifying which intermediate products (like a sauce or dough) currently contain a specific starting ingredient.
Datasets
The study utilizes the NHK Recipe Dataset ("Minna no Kyou no Ryouri"), a collection of approximately 25,000 recipes from Japan’s national broadcaster. These recipes are considered "highly clean" because they are authored by professional chefs and editors using consistent terminology. From this source, the authors manually annotated a subset of 100 recipes with full state-transition metadata. Sixty recipes were used for training and fine-tuning, while forty were reserved for the probing tasks.
Results and Discussion
The Intelligence Floor and Fine-Tuning
A key finding was the existence of an "intelligence floor" around the 10-billion parameter mark. Models with fewer parameters, such as Llama-3.1-8B, actually saw performance decreases after fine-tuning, suggesting they lack the base capacity to map linguistic instructions to complex physical states. Conversely, larger models saw massive gains; Llama-3.1-70B's accuracy on the Step Reference task jumped from 0.224 to 0.708 after fine-tuning.
Multimodality and Benchmarks
Large Multimodal Models (LMMs) like Qwen-2.5-VL-72B achieved the highest overall score (0.919), suggesting that visual context helps models "back-calculate" intermediate steps. Furthermore, the fine-tuned Swallow-3.1-70B (a Japanese-optimized model) achieved a score of 0.837, outperforming the zero-shot performance of GPT-4o (0.824) on these specific kitchen-reasoning tasks.
Conclusion
While fine-tuning significantly improves state tracking, models still struggle with "subtraction" logic (e.g., discarding water or wiping off oil) and complex splitting/recombining of ingredients. Future work will focus on expanding the annotated dataset to cover these edge cases and further refining the models' ability to maintain a consistent internal world state throughout a procedure.
Summary updated