TL;DR
This paper introduces EHFR-Net, a novel lightweight hybrid neural network that combines Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) for efficient food recognition. By developing a "Location-Preserving ViT" (LP-ViT) module that retains spatial information and integrating it with CNNs in a "Hybrid Block" (HBlock), the model achieves superior accuracy on food image datasets while maintaining a small footprint suitable for mobile devices.
What Problem Does This Paper Address?
Accurate food recognition is crucial for intelligent nutrition management and personalized dietary tracking, especially on resource-constrained mobile devices. While deep learning models like CNNs and ViTs have shown promise, existing lightweight solutions often struggle to balance accuracy with efficiency. CNNs excel at local feature extraction but are less adept at capturing long-range dependencies in food images, which often have dispersed ingredients. ViTs, on the other hand, are good at global feature extraction but can be computationally expensive and may overlook crucial local spatial details. This paper aims to bridge this gap by proposing a hybrid architecture that efficiently leverages the strengths of both CNNs and ViTs for lightweight and accurate food recognition.
Methods
The core innovation of this work lies in the Efficient Hybrid Food Recognition Net (EHFR-Net). It's built upon two key components:
- Location-Preserving Vision Transformer (LP-ViT): Traditional ViTs often discard spatial information during the initial tokenization and embedding process, requiring positional encodings to reintroduce it. LP-ViT addresses this by using "unfold" and "fold" operations to process image patches. This allows it to directly compute correlations between pixels while preserving their original spatial relationships, making it more effective for fine-grained tasks like food recognition. It also employs a "separate self-attention" mechanism, which is more efficient than standard Multi-Head Attention.
- Hybrid Block (HBlock): This is the fundamental building block of EHFR-Net. It integrates an inverted residual block (from MobileNetV2) for local feature extraction (CNN side) with the proposed LP-ViT module for global feature extraction. These local and global features are then fused through skip connections.
- Optimized Hierarchical Layout: The EHFR-Net architecture is specifically designed to leverage the characteristics of HBlock, with a reduced number of layers in certain parts to further minimize parameters and computations, leading to a lightweight model.
Datasets
The proposed EHFR-Net was evaluated on three widely-used food image recognition datasets:
- ETHZ Food-101: A large-scale dataset of Western cuisine images.
- Vireo Food-172: A dataset focusing on Chinese cuisine.
- UEC Food256: A dataset comprising Japanese dishes.
The paper does not introduce new datasets but uses these established benchmarks to demonstrate the effectiveness of their approach.
Key Insights and Ideas
- Spatial Information is Key for Food Recognition: The paper highlights that while ViTs are powerful for global context, their tendency to disregard initial spatial information can be detrimental for fine-grained tasks like food recognition, where ingredient placement matters.
- Hybridization is More Than Just Concatenation: The HBlock's design, which fuses local CNN features with global LP-ViT features through skip connections, is crucial for effectively capturing the complex visual characteristics of food.
- Lightweight Design Requires Architectural Innovation: Simply using existing lightweight modules isn't enough. The paper demonstrates that a carefully designed hierarchical network layout, tailored to the hybrid block's strengths, is essential for achieving both accuracy and efficiency.
- LP-ViT Offers a More Efficient Path to Global Features: The LP-ViT module proves to be more efficient than standard ViT approaches for extracting global features, requiring fewer computations and parameters while achieving comparable or better results.
Key Results
EHFR-Net demonstrates strong performance across the tested datasets:
- ETHZ Food-101: Achieved 90.7% top-1 accuracy with 2.8M parameters, outperforming MobileViTv2 (87.2%) with a similar parameter count by 3.5%.
- Vireo Food-172: Achieved 91.38% top-1 accuracy, showing significant improvements over CNN-based models and ViT-based SOTA models.
- UEC Food256: Achieved 71.30% top-1 accuracy, also outperforming comparable models.
- Qualitative Analysis: Grad-CAM visualizations show that EHFR-Net can better focus on distinctive features, especially for complex dishes, where traditional CNNs might fail.
Limitations and Future Work
- Latency Concerns: The paper acknowledges that while the model is lightweight in terms of parameters and FLOPs, the predominant use of ViT structures in the overall architecture can still introduce latency during inference, especially on mobile devices.
- Future Work: The authors plan to address latency by exploring reparameterization techniques during the inference stage. They also aim to optimize the model for better applicability on specific hardware and operating systems, further enhancing its utility for daily dietary and nutritional management.
Summary updated