TL;DR
This paper tackles the challenge of food image classification in real-world scenarios where new food classes emerge over time and data distributions are imbalanced. They introduce a novel framework to simulate realistic food consumption patterns and a plug-and-play module, DMU, that significantly improves existing online class-incremental learning methods by intelligently selecting training data.
What Problem Does This Paper Address?
Food image classification is crucial for applications like dietary monitoring and health analysis. Traditional models, however, are trained on static datasets with fixed classes and balanced distributions. In reality, food consumption patterns are dynamic, influenced by culture, economics, and personal preferences, leading to evolving food classes and imbalanced data. This necessitates models that can learn continuously from a stream of data without forgetting previously learned information – a problem known as catastrophic forgetting. Online Class-Incremental Learning (OCIL) aims to address this by learning from sequential data streams, but existing OCIL methods often assume unrealistic balanced data distributions, limiting their applicability to real-world scenarios like food classification.
Methods
The paper proposes two main contributions:
- Realistic Data Distribution Module (RDDM): This probabilistic framework simulates realistic food consumption patterns. It models three key characteristics of real-world data:
- Variable number of classes per task: Each task introduces new classes, but the total number of classes is not fixed.
- Imbalanced sample sizes: The number of images per class within a task can vary significantly.
-
Overlapping classes: Classes can reappear across different tasks. RDDM uses hyperparameters $\alpha$ and $\beta$ to control class repetition and the introduction of new classes, respectively, allowing for the simulation of short-term, moderate-term, and long-term consumption patterns.
-
Dynamic Model Update (DMU): This is a plug-and-play module designed to enhance existing Experience Replay (ER) based OCIL methods. DMU operates by creating two "virtual" models. One virtual model processes the entire input batch along with retrieved exemplars, while the other processes only the new class data (non-repeated samples) and exemplars. The DMU then compares the performance of these virtual models against a threshold (based on initial task accuracy) and directs the data from the better-performing branch to train the main model. This intelligent selection of training data aims to improve learning accuracy and mitigate catastrophic forgetting, especially in the presence of data repetition and imbalances.
Datasets
The authors introduce two benchmarks for their Realistic Online Class Incremental Learning (R-OCIL) setting:
- Food-101: A dataset with 101 food classes.
- VFN (Visual Food Nexus): A dataset with 74 food classes, representing commonly consumed food categories in the US.
Both datasets are used with simulated food consumption patterns generated by the RDDM module. Experiments are conducted with task sizes of 5, 10, and 20, and a memory buffer size of 5K.
Key Insights and Ideas
- Real-world data is messy: The paper strongly emphasizes that real-world food consumption patterns are inherently imbalanced and dynamic, a stark contrast to the idealized, balanced datasets often used in OCIL research.
- Intelligent data selection is key: The DMU module's success highlights that simply training on all available data (even with replay) isn't optimal. Dynamically choosing which data to focus on based on current model performance is crucial for better learning and reduced forgetting.
- Simulating realistic patterns matters: The RDDM framework demonstrates the importance of creating realistic data distributions to properly evaluate OCIL methods in practical scenarios.
- Short-term patterns are easier: The results suggest that scenarios with fewer classes and shorter learning horizons lead to better performance and less forgetting, likely due to reduced complexity.
Key Results
- The proposed DMU module consistently improves the performance of existing ER OCIL methods across various task sizes and food consumption patterns on both Food-101 and VFN datasets.
- On average, DMU provides a 26% increase in accuracy on the Food-101 dataset and a 12.5% increase on the VFN dataset compared to baseline ER methods.
- The RDDM framework successfully simulates realistic food consumption patterns, revealing that performance degrades as task sizes increase and in long-term consumption scenarios due to increased class complexity and potential for imbalance.
- Even with smaller buffer sizes (1K and 0.5K), DMU continues to show significant performance enhancements.
Limitations and Future Work
- Limitations: The paper acknowledges that the "Fine-tune" baseline, which only uses new class data, serves as a lower bound and is significantly outperformed by ER methods. The core limitation addressed is the gap between current OCIL assumptions and real-world data complexity.
- Future Work: The authors plan to explore improving the exemplar storage procedure by leveraging the most representative samples simulated by the RDDM framework. They also aim to evaluate their method on larger, open-world supervised datasets.
Summary updated