Pic2Prep: A Multimodal Conversational Agent for Cooking Assistance

Renjith Prasad Kaippilly Mana, Chathurangi Shyalika, Revathy Venkataramanan, Darssan Eswaramoorthi, Amit P. ShethAAAI 2025 · DOI

TL;DR

This paper introduces Pic2Prep, a multimodal conversational agent that generates detailed cooking instructions, ingredient lists, and actions directly from food images. By leveraging novel synthetic image generation and a pipeline of vision-language models, Pic2Prep aims to bridge the gap between visual food understanding and actionable culinary guidance, paving the way for more intelligent kitchen assistance.

What Problem Does This Paper Address?

The paper tackles the limitations of current food computation models, which often provide only basic nutritional insights and lack the depth to offer actionable guidance for complex culinary tasks. While existing vision-language models can recognize ingredients or estimate nutrition from food images, they struggle to infer cooking actions, suggest ingredient substitutions, or provide comprehensive, step-by-step instructions. This gap hinders the development of truly intelligent kitchen assistants that can go beyond simple identification to actively guide users through the cooking process.

Methods

Pic2Prep employs a three-stage multimodal framework:

  1. Synthetic Image Generation: A novel dataset of food images is created using Stable Diffusion, with recipe titles and ingredient lists from the Recipe1M dataset serving as input. This approach generates diverse visual representations to improve model generalization.
  2. Instruction and Ingredient Modeling: Two separate BLIP (Bootstrapping Language-Image Pre-training) models are fine-tuned on the synthetic dataset. One model focuses on generating cooking instructions, while the other extracts ingredient lists from food images.
  3. Action Mapping: The CookGen model is used to derive detailed cooking actions from the generated instructions. Subsequently, a custom mapper, trained using the Mistral language model, links these actions to their corresponding ingredients, creating a comprehensive understanding of the cooking process.

The system is presented through an interactive user interface that allows users to upload images and ask targeted questions.

Datasets

  • Recipe1M: Used as the source for recipe titles and ingredient lists to generate synthetic food images.
  • Newly Introduced Dataset: A novel dataset of synthesized food images generated through Stable Diffusion, specifically designed for training BLIP models to extract cooking instructions and ingredients.

Key Insights and Ideas

  • Bridging the Vision-Action Gap: The core innovation lies in connecting visual food understanding (from images) with actionable cooking guidance (instructions and actions).
  • Synthetic Data for Generalization: Using Stable Diffusion to generate a diverse synthetic dataset is a key strategy to improve the model's ability to handle real-world food images and variations.
  • Multimodal Pipeline for Comprehensiveness: The cascaded approach, from image generation to instruction and action modeling, allows for a more holistic understanding of the cooking process than single-modality models.
  • Interactive Conversational Agent: The design of an intuitive chatbot interface enhances user experience, making complex food information accessible through natural language queries.

Key Results

The paper demonstrates the functionality of Pic2Prep through its interactive interface. While specific quantitative results (e.g., accuracy metrics for instruction generation or ingredient recognition) are not detailed in the provided text, the qualitative findings highlight the system's ability to:

  • Generate detailed cooking instructions from food images.
  • Extract ingredient lists from food images.
  • Infer specific cooking actions related to ingredients and instructions.
  • Provide real-time, personalized responses to user queries.

The paper showcases examples of the system responding to queries like "What are the ingredients in this image?" and "What are the cooking actions for these instructions?".

Limitations and Future Work

Limitations:

  • The paper acknowledges that current models may restrict their scope and do not always adopt a comprehensive perspective on recipes, potentially overlooking the integration of ingredients and cooking processes.
  • The system provides a fallback response ("I don't know") if the image is unrelated to food or a query falls outside the system's scope.

Future Work:

  • Expand Pic2Prep to include more complex cooking processes.
  • Incorporate additional types of user queries.
  • Explore more advanced applications such as ingredient substitution and dietary recommendation systems.
  • Develop fully automated kitchen assistance capabilities.

Summary updated

BibTeX
@inproceedings{mana2025pic,
  author = {Renjith Prasad Kaippilly Mana and Chathurangi Shyalika and Revathy Venkataramanan and Darssan Eswaramoorthi and Amit P. Sheth},
  title = {{Pic2Prep: A Multimodal Conversational Agent for Cooking Assistance}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35359},
}
Figure