TL;DR
$\phi$-DPO (Fairness Direct Preference Optimization) is a novel framework designed to tackle the dual challenges of catastrophic forgetting and data imbalance in the continual learning of Large Multimodal Models (LMMs). By reframing continual learning through a fairness-aware preference optimization lens, it eliminates the need for explicit reward models while mathematically ensuring balanced gradient updates across skewed data distributions.
What Problem Does This Paper Address?
Large Multimodal Models (LMMs) excel at multi-task visual and textual reasoning, but updating them with new knowledge sequentially—known as Continual Learning (CL)—presents a notorious dilemma. Models suffer from catastrophic forgetting, where learning new tasks degrades performance on previously mastered ones.
Furthermore, real-world multimodal datasets are highly imbalanced. As an LMM encounters sequential tasks, it faces shifts in both visual distributions and alignment objectives (e.g., moving from scientific diagrams to object localization, as shown in Figure 3).
Within these tasks, topic distributions are often heavily skewed (e.g., a massive amount of biology data but very few examples for grammar or word study, as shown in Figure 2). Traditional CL methods like Low-Rank Adaptation (LoRA) or Knowledge Distillation (KD) inherit or amplify these biases, leading to gradient updates that favor majority classes and tank performance on minority "tail" distributions.
Methods
The authors reframe continual learning as a Reinforcement Learning from Human Feedback (RLHF) problem, where the goal is to maximize a reward for memory retention and adaptability while penalizing drift from the previous model state via a KL-divergence constraint. To bypass the complex and expensive process of training an explicit reward model, they leverage Direct Preference Optimization (DPO). In this setup, a preferred response ($y^+$) represents a well-retained/adapted output, while a dispreferred response ($y^-$) represents a forgotten or hallucinated output.
To address data imbalance, the authors introduce Fairness DPO ($\phi$-DPO). Standard DPO gradients are easily dominated by majority groups. Inspired by focal loss, $\phi$-DPO introduces a modulating factor $(1 - p(z))^\gamma$ to the standard DPO loss function:
$$L^\gamma_{\text{DPO}}(\theta; \mu) = - \mathbb{E}_{z\sim\mu} \left[ (1 - p(z))^\gamma \log p(z) \right]$$
Where $\gamma$ is a focusing parameter. This factor down-weights easy, well-learned examples (typically from majority classes) and forces the model to focus on hard, underrepresented preference pairs.
Datasets
The framework was evaluated across three major LMM continual learning benchmark suites:
- CoIN: An eight-task benchmark spanning ScienceQA, ImageNet, VizWiz, Grounding, TextVQA, GQA, VQAv2, and OCR-VQA.
- MLLM-CL Domain: Five sequential domains including Remote Sensing, Medical, Autonomous Driving, Science, and Finance.
- MLLM-CL Ability: Four task-incremental domains including OCR, Math & Logic, Visual Perception, and GUI Agent.
The DPO Label Contribution: Because these standard benchmarks lack the pairwise preference annotations required for DPO, the authors constructed a new dataset. They utilized a large language model to generate plausible but subtly flawed alternative responses ($y^-$) conditioned on the ground-truth answer ($y^+$), followed by rigorous human verification.
Key Insights and Ideas
- DPO as Bounded Distillation: The authors provide a theoretical proof demonstrating that traditional Knowledge Distillation (KD) loss is mathematically bounded by the DPO loss. This positions DPO as a generalized, more robust form of distillation that selectively amplifies high-reward responses rather than just matching output probabilities.
- Mathematical Guarantee of Fairness: The paper proves that as the focusing parameter $\gamma$ increases, the gradient discrepancy between an ideal balanced data distribution and a practical imbalanced distribution vanishes ($\lim_{\gamma\to\infty} \|B_\gamma(\theta)\| = 0$).
- Controlling Stability vs. Plasticity: The Lagrangian multiplier $\beta$ acts as a precise dial for the stability-plasticity trade-off; higher values enforce stability (keeping the model close to its predecessor), while lower values grant the plasticity needed to adapt to new tasks.
Key Results
$\phi$-DPO achieved State-of-the-Art (SoTA) metrics across all evaluated benchmarks:
- Minimal Forgetting: On the MLLM-CL Domain benchmark, $\phi$-DPO achieved a Mean Final Accuracy (MFN) of 74.00% and a Backward Transfer (BWT) of -0.37%, drastically outperforming standard LoRA fine-tuning (BWT of -14.97%) and vanilla DPO (BWT of -1.33%).
- Robustness to Skewed Tasks: On the CoIN benchmark, it achieved substantial gains in vision-centric tasks, hitting 95.61% on ImageNet and 68.69% on OCR, while maintaining a high Mean Average Accuracy (MAA) of 74.94%.
- Optimal Hyperparameters: Ablation studies revealed that $\gamma = 2.00$ and $\beta = 0.10$ provided the best trade-off, mitigating data imbalance without triggering gradient vanishing.
Limitations and Future Work
While highly effective, the authors note that $\phi$-DPO relies on a fixed set of hyperparameters ($\beta$ and $\gamma$) that require manual tuning. Additionally, the quality of the framework is highly sensitive to the stability of the generated DPO preference pairs. If the automated LLM generation phase introduces misleading pairwise correlations due to severe domain shifts or model uncertainty, it can yield suboptimal distillation targets. The authors point toward adaptive DPO data generation strategies as a promising avenue for future research.
Summary updated