TL;DR
$\phi$-DPO introduces a novel fairness-aware Direct Preference Optimization framework designed for continual learning in Large Multimodal Models (LMMs). By framing knowledge retention as a pairwise preference alignment problem and introducing a focal-loss-inspired gradient modulator, it simultaneously prevents catastrophic forgetting and mitigates the performance biases caused by severely imbalanced training data.
What Problem Does This Paper Address?
When Large Multimodal Models (LMMs) are sequentially trained on new tasks or domains—a paradigm known as Continual Learning (CL)—they suffer from catastrophic forgetting, where newly acquired knowledge overwrites previously learned skills.
While prior work has focused heavily on forgetting, this paper highlights a second, deeply intertwined challenge: data imbalance. Real-world multimodal datasets are rarely uniform; they exhibit severe topic and modality skew. For example, a model learning sequential tasks might transition from structured scientific diagrams to object localization, and finally to text extraction.
Standard fine-tuning techniques like LoRA or Knowledge Distillation (KD) inherit these dataset biases. Gradients become heavily skewed toward majority classes or dominant modalities, which not only degrades performance on minority "tail" classes but also accelerates the catastrophic forgetting of earlier, less-represented tasks.
Methods
The authors propose Fairness Direct Preference Optimization ($\phi$-DPO), which tackles both forgetting and unfairness through a unified preference-learning objective.
- Continual Learning via DPO: Instead of using computationally heavy Reinforcement Learning from Human Feedback (RLHF) or traditional Knowledge Distillation, the authors reformulate continual learning as a Direct Preference Optimization (DPO) problem. They define a preference pair for each instruction: a "preferred" output $y^+$ (representing well-retained or well-adapted memory) and a "dispreferred" output $y^-$ (representing forgotten or hallucinated behavior). Maximizing this preference implicitly regularizes the model, keeping its updates bounded within the information manifold of the previous learning step.
-
The $\phi$-DPO Loss Function: Vanilla DPO still suffers under imbalanced data because majority groups dominate the gradient updates. To fix this, the authors introduce a focusing parameter $\gamma$ (inspired by focal loss) into the DPO objective: $$\mathcal{L}_{\text{DPO}}^\gamma(\theta; \mu) = - \mathbb{E}_{z\sim\mu} \left[ (1 - p(z))^\gamma \log p(z) \right]$$ Here, $w^\gamma_k(\theta)$ acts as a modulating factor that dynamically down-weights the gradients of easy, majority-class examples and amplifies the gradients of hard, minority-class examples. The authors mathematically prove that as $\gamma$ becomes sufficiently large, the gradient updates balance out across all groups regardless of the underlying data bias.
-
Parameter Efficiency: To ensure the framework is practical and memory-efficient, the LMM is optimized using Low-Rank Adaptation (LoRA) adapters rather than updating the full backbone parameters.
Datasets
The framework was evaluated across three major continual learning benchmark suites:
- CoIN: Comprising eight diverse vision-language tasks including ScienceQA, ImageNet, VizWiz, Grounding, TextVQA, GQA, VQAv2, and OCR-VQA.
- MLLM-CL Domain: A domain-incremental benchmark spanning five sequential domains: Remote Sensing, Medical, Autonomous Driving, Science, and Finance.
- MLLM-CL Ability: A task-incremental benchmark covering OCR, Math & Logic, Visual Perception, and GUI Agent tasks.
New Contribution: Because standard CL benchmarks lack the pairwise preference annotations required for DPO, the authors constructed a brand-new preference dataset. They used a large language model to generate plausible, subtle hallucinations ($y^-$) conditioned on the original prompt and correct answer ($y^+$), followed by rigorous human verification by five specialists over a month to filter out poor annotations.
Key Insights and Ideas
- DPO as a Generalized Distillation: The authors provide a two-sided theoretical proof showing that the traditional Knowledge Distillation (KD) loss is strictly bounded by the DPO loss. This means DPO inherently retains the regularization benefits of KD while adding an adaptive, pairwise preference mechanism that selectively amplifies high-reward responses.
- Gradient Balancing via Sample Difficulty: The core "aha" moment of the paper is that fairness under data imbalance can be treated as a sample-difficulty problem. By penalizing the model less for mistakes on rare classes and dynamically scaling down gradients for over-represented classes, the model updates remain stable and unbiased across sequential tasks.
Key Results
$\phi$-DPO achieved State-of-the-Art (SoTA) performance across all evaluated benchmarks, handily outperforming prior methods like MoELoRA, DISCO, and MR-LoRA.
- MLLM-CL Domain: Achieved a Mean Finetune Accuracy (MFT) of 74.29% and a Mean Final Accuracy (MFN) of 74.00%. Crucially, it demonstrated minimal forgetting with a Backward Transfer (BWT) score of just -0.37% (compared to standard LoRA-FT at -14.97%).
- CoIN Benchmark: Consistently outperformed prior methods on vision-centric tasks, hitting 95.61% on ImageNet and 68.69% on OCR, while maintaining a high Mean Average Accuracy (MAA) of 74.94%.
- Noise Robustness: In ablation studies where controlled noise was injected into the preference data, $\phi$-DPO maintained stable performance, proving its robustness against imperfect, real-world annotations.
Limitations and Future Work
Despite its strong performance, the authors note a few limitations:
- Hyperparameter Sensitivity: The framework relies on manual tuning of fixed hyperparameters, specifically the divergence multiplier $\beta$ and the focusing parameter $\gamma$. Improper tuning can lead to gradient vanishing or a flat loss landscape, limiting plasticity.
- Label Stability: The quality of the DPO training is highly sensitive to the stability of the generated preference pairs. Severe domain shifts or high model uncertainty during the negative-pair generation phase can still lead to suboptimal distillation targets.
Future work will focus on developing more robust, adaptive DPO strategies that can dynamically adjust these hyperparameters during the continual learning process.
Summary updated