Abstract
Domain Adaptation of Black-Box Predictors (DABP) is a challenging task where a target model must be adapted using only unlabeled target data and predictions from a source black-box API. A significant hurdle in current DABP methods is "minority class forgetting," where biased source predictions cause the target model to ignore classes with few samples, often leading to 0% accuracy for those categories. This paper introduces Reviewing the Forgotten Classes (RFC), a framework that integrates an adaptive Selection Training (ST) module with Neighborhood Clustering (NC). RFC identifies neglected classes through prediction entropy and employs a noise-robust training strategy to "review" them. Our results across major benchmarks like Office-31 and Office-Home show that RFC significantly reduces forgotten classes and achieves state-of-the-art accuracy.
The Minority Class Forgetting Problem
In DABP, researchers typically rely on Knowledge Distillation (KD) to transfer knowledge from a source black-box to a target model. However, domain shift often biases the source model's predictions, creating an imbalance. KD tends to amplify this bias, causing the target model to "push" samples from minority classes into majority ones. This leads to a phenomenon where certain classes are completely forgotten by the model, dropping to exactly 0% accuracy.
Proposed Framework: RFC
The RFC framework addresses forgetting through two main components:
- Selection Training (ST): This module acts as a "review" mechanism. It adaptively selects classes likely to be forgotten by monitoring both prediction frequency and prediction entropy. High entropy indicates model confusion, signaling a need for more focused training. To handle noisy labels from the black-box, ST uses a small-loss criterion based on an Exponential Moving Average (EMA) to select clean samples. These samples undergo noise-robust training with strong data augmentation and a specialized loss function (Reverse Cross-Entropy).
- Neighborhood Clustering (NC): To counter the bias-reinforcing nature of standard KD, RFC employs Neighborhood Clustering. This encourages the model to learn balanced features by ensuring that local neighbors in the feature space share similar predictions, promoting a more globally consistent semantic structure.
Experimental Results
RFC was evaluated on Office-31, Office-Home, and VisDA-17 benchmarks. It consistently outperformed existing state-of-the-art methods. On Office-31, RFC achieved 89.2% average accuracy, notably fixing the 0% accuracy issue for several minority classes.
As shown in Figure 6, the ST module is highly effective as a "plug-and-play" component. When added to existing methods like DINE or BETA, it immediately prevents class collapse. RFC, combining ST with NC, provides the most stable and highest performance, effectively "reviewing" the classes that other methods forget.
Conclusion
The RFC framework provides a robust solution to the minority class forgetting problem in black-box domain adaptation. By adaptively "reviewing" neglected classes and leveraging neighborhood consistency, RFC achieves superior performance and ensures a more balanced adaptation. Future work will explore extending this approach to open-set scenarios and improving the initial "warm-up" phase for complex, large-scale datasets.
Summary updated