SFOOD: A Multimodal Benchmark for Comprehensive Food Attribute Analysis Beyond RGB with Spectral Insights

Zhenbo Xu, Jinghan Yang, Gong Huang, Jiqing Feng, Liu Liu, Ruihan Sun, Ajin Meng, Zhuo Zhang, Zhaofeng HearXiv 2025 · DOI · arXiv

Introduction

While computer vision has achieved significant milestones in general object recognition, food analysis remains a formidable challenge. Current datasets predominantly focus on basic classification (e.g., distinguishing a pizza from a taco), failing to capture the multi-dimensional physical and chemical attributes essential for real-world food intelligence. Properties such as precise weight, nutritional content, and internal chemical markers like sweetness (Brix) are often "invisible" to standard RGB sensors.

To bridge this "Food Gap," we introduce SFOOD (Spectral FOOD), a massive multimodal benchmark comprising 2.35 million data points across 3,266 subcategories. SFOOD transcends traditional visual recognition by integrating hyperspectral imaging, precision weight measurements, and digital sweetness levels.

Overview of SFOOD benchmark tasks including counting, VQA, recognition, segmentation, and nutrition analysis.

Methodology and Dataset Composition

The SFOOD benchmark was developed through a rigorous four-stage pipeline designed to ensure data quality and diversity:

  1. Data Sourcing & Filtering: Researchers aggregated 11 existing open-source datasets (including Food-101, Food2K, and Nutrition5K) and unified them into a standardized taxonomy.
  2. Multimodal Acquisition: To capture data beyond the visible spectrum, the team utilized dual hyperspectral cameras (covering 400nm to 1020nm), precision scales accurate to 0.01g, and digital sweetness meters.
  3. Intelligent Labeling: A hybrid approach combined expert manual annotation with AI-assisted labeling using models like DINOv2 and Qwen2-VL to provide high-quality instance segmentation and category alignment.
  4. Weight-Adaptive Splitting: Instead of random partitioning, a model-based difficulty scorer was used to ensure the test set contains a higher concentration of challenging, real-world samples.
Examples of fine-grained food recognition categories within the SFOOD dataset, illustrating the high intra-class variance.

Key Insights and Results

The "Digital Blindness" of LVLMs

A primary finding of the study is that top-tier Large Vision Language Models (LVLMs) like GPT-4o and Gemini are surprisingly poor at "digitizing" food. While these models can generate complex recipes, they frequently fail at estimating physical properties. In weight estimation tasks, GPT-4o and Gemini exhibited failure or refusal rates as high as 93.5% and 98%, respectively. This suggests that current AI lacks the grounded physical understanding necessary for precise food analysis.

The Spectral Advantage

The research demonstrates that spectral data is a "silver bullet" for tasks where RGB fails. In herbal medicine classification—where different herbs often look identical to the human eye—moving from RGB to 40-band spectral data improved Top-1 accuracy from a dismal 29.28% to 70.98%. Similarly, hyperspectral data significantly reduced the Mean Absolute Error (MAE) in sweetness (Brix) prediction compared to RGB-based estimates.

Benchmark Difficulty

SFOOD proves to be a much tougher testbed than previous datasets. Models like the Swin Transformer, which achieve ~83% accuracy on ImageNet, drop to ~48% on SFOOD. This highlights the extreme intra-class variance in food, where the same dish can appear vastly different based on preparation, lighting, or portioning.

Conclusion

SFOOD shifts the paradigm from simple food "recognition" to comprehensive "digitization." By providing a large-scale, multimodal dataset with physical and chemical ground truths, this benchmark enables the development of AI capable of extracting precise properties from a distance—a critical step for future applications in nutrition, smart agriculture, and medicine.

Reference: Xu, Z., et al. (2025). SFOOD: A Multimodal Benchmark for Comprehensive Food Attribute Analysis Beyond RGB with Spectral Insights. arXiv:2507.04412v1 [cs.CV].

Summary updated

Abstract

With the rise and development of computer vision and LLMs, intelligence is everywhere, especially for people and cars. However, for tremendous food attributes (such as origin, quantity, weight, quality, sweetness, etc.), existing research still mainly focuses on the study of categories. The reason is the lack of a large and comprehensive benchmark for food. Besides, many food attributes (such as sweetness, weight, and fine-grained categories) are challenging to accurately percept solely through RGB cameras. To fulfill this gap and promote the development of intelligent food analysis, in this paper, we built the first large-scale spectral food (SFOOD) benchmark suite. We spent a lot of manpower and equipment costs to organize existing food datasets and collect hyperspectral images of hundreds of foods, and we used instruments to experimentally determine food attributes such as sweetness and weight. The resulting benchmark consists of 3,266 food categories and 2,351 k data points for 17 main food categories. Extensive evaluations find that: (i) Large-scale models are still poor at digitizing food. Compared to people and cars, food has gradually become one of the most difficult objects to study; (ii) Spectrum data are crucial for analyzing food properties (such as sweetness). Our benchmark will be open source and continuously iterated for different food analysis tasks.
BibTeX
@misc{xu2025sfood,
  author = {Zhenbo Xu and Jinghan Yang and Gong Huang and Jiqing Feng and Liu Liu and Ruihan Sun and Ajin Meng and Zhuo Zhang and Zhaofeng He},
  title = {{SFOOD: A Multimodal Benchmark for Comprehensive Food Attribute Analysis Beyond RGB with Spectral Insights}},
  year = {2025},
  eprint = {2507.04412},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.04412},
}
Figure