Introduction
Large Multimodal Models (LMMs) have revolutionized AI-driven food analysis, enabling complex interactions ranging from simple ingredient identification to detailed recipe generation and nutritional estimation. However, as these models are tasked with learning new culinary skills sequentially, they frequently encounter "catastrophic forgetting"—a phenomenon where new knowledge overwrites previously learned information. This paper introduces Dual-LoRA, a framework designed to facilitate continual learning across diverse food-related tasks without sacrificing prior performance.
The Challenge of Heterogeneous Food Tasks
Food analysis involves heterogeneous tasks: some require structured lists (ingredients), while others demand long-form generative text (recipes). Standard Continual Learning (CL) methods often struggle with this diversity, especially when task labels are unavailable during inference. In a typical three-task sequence, a standard model's performance on the first task often plummets as soon as the second task is introduced. For example, in a sequence of Ingredients → Recipes → Nutrition, a standard LLaVA-LoRA baseline saw its ingredient recognition IoU drop from 36.95 to 12.73 after learning the second task.
Methodology: Dual-LoRA and Pseudo Replay
The proposed framework addresses forgetting through a two-pronged strategy: a novel adapter architecture and a self-correcting data replay mechanism.
1. Dual-LoRA Architecture
The architecture utilizes two distinct Low-Rank Adapters (LoRAs) for each task:
- Specialized LoRA: Focuses on task-specific nuances. It employs orthogonal regularization to ensure that updates for new tasks occur in a mathematical subspace that does not interfere with the parameters of previous tasks.
- Cooperative LoRA: Designed to capture and share common culinary knowledge across tasks, such as the relationship between specific ingredients and cooking methods.
2. Quality-Enhanced Pseudo Replay
To maintain old knowledge without storing massive historical datasets, the model generates "pseudo-samples" of past tasks. To mitigate the risk of hallucinations common in LMMs, the authors implemented a quality filter:
- Ingredient Consistency: Uses majority voting across multiple generations to ensure accuracy.
- Recipe Similarity: Employs Sentence-BERT to select the most semantically consistent recipe version from several generated candidates.
Experimental Results
Evaluated on the Uni-Food dataset—a comprehensive benchmark of nearly 100,000 food images—the Dual-LoRA framework demonstrated superior stability compared to previous state-of-the-art methods like O-LoRA.
- Performance Stability: Dual-LoRA maintained a stable ingredient recognition score of 36.99 even after completing the third task in the sequence.
- Generative Performance: Recipe generation metrics (SacreBLEU and Rouge_L) remained robust even after the model learned nutrition estimation, showing significantly less degradation than baseline models.
- Efficiency: The framework adds only ~0.2% more parameters per task and eliminates the need for raw data storage, though it increases training time by approximately 1.3x due to the pseudo-sample generation phase.
Conclusion
Dual-LoRA effectively balances task-specific separation with cross-task knowledge sharing. By leveraging internal consistency for quality control, it provides a scalable, task-agnostic solution for lifelong learning in multimodal domains. Future work will explore scaling this approach to a broader range of tasks and optimizing the computational overhead of the pseudo-replay phase.
Summary updated