Calorie Estimation in a Real-World Recipe Service

Jun Harashima, Makoto Hiramatsu, Satoshi SanjoAAAI 2020 · DOI

Abstract

Estimating nutritional content in user-generated recipes is a significant challenge due to the unstructured nature of the data. This paper presents a machine-learning-based system developed by Cookpad Inc. to automate calorie estimation. The approach decomposes the problem into two primary subtasks: ingredient normalization and serving size estimation. By deploying these models into a production environment with a human-in-the-loop feedback cycle, the system has successfully estimated calories for over 100,000 real-world recipes.

The Challenge of Unstructured Recipe Data

While calorie estimation from food photography is a popular research area, calculating calories directly from text recipes is surprisingly complex. The fundamental formula for calorie estimation is straightforward:

$$c(r) = \frac{\sum (\text{calories per 100g} \times \text{grams of ingredient} / 100)}{\text{number of servings}}$$

However, user-generated content rarely follows a strict schema. As shown in the annotated recipe below, users often include descriptive text within ingredient names (e.g., "Sesame oil (for finishing)") and use ambiguous serving sizes like "14 pieces" or "2 small bowls" instead of standard counts.

An example of a user-generated recipe with annotated components highlighting the lack of structure in 'amount' and 'ingredients' fields.

Methodology

The authors addressed these challenges by framing them as two distinct machine learning problems.

1. Ingredient Normalization

To map raw strings like "carrot (sliced)" to a canonical nutritional dictionary entry, the team used a sequence-to-sequence encoder-decoder model based on LSTMs. Interestingly, they found that attention mechanisms did not improve performance for the short strings typical of ingredients and opted for a simpler architecture to reduce deployment complexity. Character-based segmentation proved more robust than subword-based methods for handling typos and script variations (Hiragana, Katakana, and Kanji).

2. Serving Estimation

Rather than predicting a continuous number, serving estimation was treated as a classification task with 20 categories (representing 1 to 20 servings). This approach better handles the discrete nature of portions (e.g., a recipe for 14 cookies likely serves 7 people). The models were tested using recipe titles, amount strings, or both as inputs.

System Architecture and Deployment

A key feature of the Cookpad system is its "human-in-the-loop" design. Because the data is inherently ambiguous, the system allows for manual annotations and corrections that are fed back into the training set, enabling continuous model improvement.

The production system architecture, illustrating the workflow between manual annotation, GPU-based training, and CPU-based inference for the live service.

Key Results and Insights

  • Normalization Performance: The ML-based model achieved 73% accuracy, a massive improvement over the 20% accuracy of a traditional regular expression baseline. It successfully handled orthographic variants and minor spelling errors.
  • Serving Estimation Performance: The model reached 63% accuracy. While lower than normalization, this is significant because human experts only agree on serving sizes between 55% and 69% of the time, suggesting the model is performing near the human ceiling.
  • The Importance of "Amount": In serving estimation, the "amount" string (e.g., "14 pieces") was found to be a much stronger predictor than the recipe title.

Limitations and Future Work

The current system faces challenges with "hallucinations" in the generative normalization model and "context blindness" in serving estimation. For example, estimating servings for "1 iron plate" of cookies requires knowing the total volume of ingredients used, which the current model does not yet incorporate. Future work will focus on integrating the full ingredient list into the serving estimation logic and expanding the training datasets to improve multi-source model performance.

Summary updated

BibTeX
@inproceedings{harashima2020calorie,
  author = {Jun Harashima and Makoto Hiramatsu and Satoshi Sanjo},
  title = {{Calorie Estimation in a Real-World Recipe Service}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i08.7041},
}
Figure