A Lightweight Hybrid Model with Location-Preserving ViT for Efficient Food Recognition

Guorui Sheng, Weiqing Min, Xiangyi Zhu, Liang Xu, Qingshuo Sun, Yancun Yang, Lili Wang, Shuqiang JiangNutrients 2024 · DOI

TL;DR

This paper introduces EHFR-Net, a novel lightweight hybrid neural network that combines Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) for efficient food recognition. By developing a "Location-Preserving ViT" (LP-ViT) module that retains spatial information and integrating it with CNNs in a "Hybrid Block" (HBlock), the model achieves superior accuracy on food image datasets while maintaining a small footprint suitable for mobile devices.

What Problem Does This Paper Address?

Accurate food recognition is crucial for intelligent nutrition management and personalized dietary tracking, especially on resource-constrained mobile devices. While deep learning models like CNNs and ViTs have shown promise, existing lightweight solutions often struggle to balance accuracy with efficiency. CNNs excel at local feature extraction but are less adept at capturing long-range dependencies in food images, which often have dispersed ingredients. ViTs, on the other hand, are good at global feature extraction but can be computationally expensive and may overlook crucial local spatial details. This paper aims to bridge this gap by proposing a hybrid architecture that efficiently leverages the strengths of both CNNs and ViTs for lightweight and accurate food recognition.

Methods

The core innovation of this work lies in the Efficient Hybrid Food Recognition Net (EHFR-Net). It's built upon two key components:

  • Location-Preserving Vision Transformer (LP-ViT): Traditional ViTs often discard spatial information during the initial tokenization and embedding process, requiring positional encodings to reintroduce it. LP-ViT addresses this by using "unfold" and "fold" operations to process image patches. This allows it to directly compute correlations between pixels while preserving their original spatial relationships, making it more effective for fine-grained tasks like food recognition. It also employs a "separate self-attention" mechanism, which is more efficient than standard Multi-Head Attention.
  • Hybrid Block (HBlock): This is the fundamental building block of EHFR-Net. It integrates an inverted residual block (from MobileNetV2) for local feature extraction (CNN side) with the proposed LP-ViT module for global feature extraction. These local and global features are then fused through skip connections.
  • Optimized Hierarchical Layout: The EHFR-Net architecture is specifically designed to leverage the characteristics of HBlock, with a reduced number of layers in certain parts to further minimize parameters and computations, leading to a lightweight model.

Datasets

The proposed EHFR-Net was evaluated on three widely-used food image recognition datasets:

  • ETHZ Food-101: A large-scale dataset of Western cuisine images.
  • Vireo Food-172: A dataset focusing on Chinese cuisine.
  • UEC Food256: A dataset comprising Japanese dishes.

The paper does not introduce new datasets but uses these established benchmarks to demonstrate the effectiveness of their approach.

Key Insights and Ideas

  • Spatial Information is Key for Food Recognition: The paper highlights that while ViTs are powerful for global context, their tendency to disregard initial spatial information can be detrimental for fine-grained tasks like food recognition, where ingredient placement matters.
  • Hybridization is More Than Just Concatenation: The HBlock's design, which fuses local CNN features with global LP-ViT features through skip connections, is crucial for effectively capturing the complex visual characteristics of food.
  • Lightweight Design Requires Architectural Innovation: Simply using existing lightweight modules isn't enough. The paper demonstrates that a carefully designed hierarchical network layout, tailored to the hybrid block's strengths, is essential for achieving both accuracy and efficiency.
  • LP-ViT Offers a More Efficient Path to Global Features: The LP-ViT module proves to be more efficient than standard ViT approaches for extracting global features, requiring fewer computations and parameters while achieving comparable or better results.

Key Results

EHFR-Net demonstrates strong performance across the tested datasets:

  • ETHZ Food-101: Achieved 90.7% top-1 accuracy with 2.8M parameters, outperforming MobileViTv2 (87.2%) with a similar parameter count by 3.5%.
  • Vireo Food-172: Achieved 91.38% top-1 accuracy, showing significant improvements over CNN-based models and ViT-based SOTA models.
  • UEC Food256: Achieved 71.30% top-1 accuracy, also outperforming comparable models.
  • Qualitative Analysis: Grad-CAM visualizations show that EHFR-Net can better focus on distinctive features, especially for complex dishes, where traditional CNNs might fail.

Limitations and Future Work

  • Latency Concerns: The paper acknowledges that while the model is lightweight in terms of parameters and FLOPs, the predominant use of ViT structures in the overall architecture can still introduce latency during inference, especially on mobile devices.
  • Future Work: The authors plan to address latency by exploring reparameterization techniques during the inference stage. They also aim to optimize the model for better applicability on specific hardware and operating systems, further enhancing its utility for daily dietary and nutritional management.

Summary updated

Abstract

Food-image recognition plays a pivotal role in intelligent nutrition management, and lightweight recognition methods based on deep learning are crucial for enabling mobile deployment. This capability empowers individuals to effectively manage their daily diet and nutrition using devices such as smartphones. In this study, we propose an Efficient Hybrid Food Recognition Net (EHFR-Net), a novel neural network that integrates Convolutional Neural Networks (CNN) and Vision Transformer (ViT). We find that in the context of food-image recognition tasks, while ViT demonstrates superiority in extracting global information, its approach of disregarding the initial spatial information hampers its efficacy. Therefore, we designed a ViT method termed Location-Preserving Vision Transformer (LP-ViT), which retains positional information during the global information extraction process. To ensure the lightweight nature of the model, we employ an inverted residual block on the CNN side to extract local features. Global and local features are seamlessly integrated by directly summing and concatenating the outputs from the convolutional and ViT structures, resulting in the creation of a unified Hybrid Block (HBlock) in a coherent manner. Moreover, we optimize the hierarchical layout of EHFR-Net to accommodate the unique characteristics of HBlock, effectively reducing the model size. Our extensive experiments on three well-known food image-recognition datasets demonstrate the superiority of our approach. For instance, on the ETHZ Food-101 dataset, our method achieves an outstanding recognition accuracy of 90.7%, which is 3.5% higher than the state-of-the-art ViT-based lightweight network MobileViTv2 (87.2%), which has an equivalent number of parameters and calculations.
BibTeX
@inproceedings{sheng2024lightweight,
  author = {Guorui Sheng and Weiqing Min and Xiangyi Zhu and Liang Xu and Qingshuo Sun and Yancun Yang and Lili Wang and Shuqiang Jiang},
  title = {{A Lightweight Hybrid Model with Location-Preserving ViT for Efficient Food Recognition}},
  booktitle = {Nutrients},
  year = {2024},
  doi = {10.3390/nu16020200},
}
Figure