This paper, presented at NeurIPS 2022, introduces ORIENT, a framework designed to make Supervised Domain Adaptation (SDA) significantly faster and more efficient by intelligently selecting which data to train on.
TL;DR
ORIENT uses Submodular Mutual Information (SMI) to select a small, highly relevant subset of source data that closely matches the target domain. By training on just 30% of the source data, the framework achieves a 3x speedup while maintaining or even improving classification accuracy compared to training on the full dataset.
The Efficiency Gap in Domain Adaptation
Deep learning models are notoriously data-hungry. When we lack sufficient labeled data for a specific "target" task (e.g., identifying medical anomalies), we often use "source" data from a related field—a process known as Domain Adaptation.
However, Supervised Domain Adaptation (SDA)—where we have a tiny amount of labeled target data to guide the model—is computationally expensive. The authors point out a striking inefficiency: training a ResNet50 model on the relatively small Office-Home dataset using state-of-the-art SDA methods can take over 18 hours, whereas training the same model on the much larger CIFAR-100 dataset takes only 14 hours. This "efficiency gap" limits the use of SDA in resource-constrained environments and increases the carbon footprint of AI research.
Methods: The ORIENT Framework
The core innovation is ORIENT (Optimized Representative Information for Efficient Network Training). Instead of training on the entire source dataset, ORIENT picks a subset that is most "useful" for the target task.
- Submodular Mutual Information (SMI): The authors use SMI functions to find a subset of source data that is both diverse (covers different features) and similar to the target data.
- Gradient-Based Similarity: To measure how "similar" two data points are, ORIENT doesn't just look at pixels. It looks at the gradients of the model's loss function. If two images produce similar gradients, the model "perceives" them similarly.
- Adaptive Selection: The subset isn't fixed. Every few epochs ($L$), the framework re-calculates which source images are currently most relevant to the target domain based on the model's current state and updates the training set.
- Theoretical Unification: A major contribution of the paper is proving that several existing subset selection methods (like GLISTER, GRAD-MATCH, and CRAIG) are actually specific mathematical instances of the SMI functions used in ORIENT.
Datasets and Evaluation
The authors evaluated ORIENT on two standard benchmarks for domain shift:
- Office-31: Contains 4,110 images across 31 categories from three domains: Amazon (product images), DSLR (high-res photos), and Webcam (low-res photos).
- Office-Home: A more challenging dataset with 10,812 images across 65 categories from four domains: Art, Clipart, Product, and Real-world.
In these tests, the "target" domain was provided with as few as 2 labeled examples per class.
Key Insights
- Quality Over Quantity: The results suggest that training on the entire source dataset can actually be counter-productive. Source data often contains "noise" or features that don't exist in the target domain. By filtering for the most relevant 30%, ORIENT removes this distraction.
- Gradients as Feature Extractors: Using the gradients of the last layer of the neural network serves as a powerful, task-specific similarity metric that evolves as the model learns.
- Better Feature Localization: Using GradCam (a visualization tool), the authors showed that models trained with ORIENT focus more accurately on the actual objects in an image, whereas models trained on the "Full" dataset often get distracted by background elements.
Key Results
- 3x Speedup: ORIENT consistently reduced training time by roughly 66-70% by using only a 0.3 fraction of the source data.
- Superior Accuracy: In many cases, ORIENT outperformed the "Full" training baseline. For example, on the Office-Home "Art to Product" task, ORIENT achieved 78% accuracy, while training on the full source dataset only reached 74%.
- Robustness: The framework proved effective when combined with various existing SDA losses, such as CCSA and $d$-SNE, showing it is a versatile "plug-and-play" module for efficiency.
Limitations and Future Work
- Memory Overhead: To select the best subset, ORIENT must store a "similarity kernel" (a large matrix of similarity scores). For very large datasets, this requires significant GPU/RAM memory, which might be a bottleneck for low-end hardware.
- Hyperparameter Sensitivity: The user must still choose the subset size ($b$) and the selection interval ($L$). While the paper provides guidelines, these may require tuning for new types of data.
- Unsupervised Settings: The current work focuses on Supervised adaptation. A natural next step is extending this logic to Unsupervised Domain Adaptation (UDA), where the target data has no labels at all.
Summary updated