Summary
This research introduces a framework for self-supervised learning (SSL) that effectively leverages data from multiple related domains, such as photos, sketches, and cartoons. By employing mutual information constraints to balance domain-invariant features with domain-specific details, the authors produce robust, unbiased representations that outperform models trained on single datasets or naively combined data.
The Problem: Dataset Bias and the Domain Gap
Most SSL methods are trained on single, large-scale datasets like ImageNet. However, these datasets are inherently biased, reflecting specific lighting, camera angles, and styles that do not represent the full "visual world."
Simply combining multiple datasets (a strategy known as "DeepAll") often leads to a drop in performance. This occurs because the model becomes confused by the "domain gap"—the stylistic differences between, for instance, a photo of a dog and a sketch of a dog. The core challenge lies in learning the high-level concept of an object without being distracted by the stylistic noise of its source domain.
Methodology: The Push-Pull of Information
The authors propose a multi-domain learning (MDL) approach governed by two competing information-theoretic constraints:
- Inter-domain MI Minimization (Invariance): The model is encouraged to minimize the Mutual Information (MI) between the learned representation ($z$) and the domain label ($d$). This forces the network to discard "unwanted variations" like style or lighting, making the representation invariant across domains. This is implemented via adversarial training using a Gradient Reversal Layer (GRL).
- Intra-domain MI Maximization (Preservation): To prevent the model from losing useful features by becoming too invariant, the authors maximize the MI between the input image ($x$) and its representation ($z$) for each domain. This ensures the model captures essential semantic details. This is implemented using the Jensen-Shannon Divergence (JSD) as a tractable lower bound of MI.
The final objective is a minimax game where the encoder satisfies the SSL task (e.g., predicting image rotation) while balancing these two constraints.
Datasets and Evaluation
The method was evaluated on three distinct setups:
- PACS: A standard multi-domain benchmark containing four domains: Photo, Art painting, Cartoon, and Sketch. It features 7 shared categories across very different styles.
- ImageNet & Places: Two massive datasets treated as separate domains (object-oriented vs. scene-oriented) to test scalability.
- PASCAL VOC 2007: Used as a downstream "target" dataset to test how well the multi-domain representations transfer to new tasks.
Key Results and Insights
- Naive Mixing is Suboptimal: Simply concatenating datasets often leads to worse performance than single-domain training because the model overfits to domain-specific biases.
- PACS Performance: On the PACS dataset, the proposed method (DeepAll+MI) improved average linear classification accuracy by 1.1% on conv5 and 3.0% on fc7 layers compared to the naive DeepAll approach.
- Large-Scale Gains: When combining ImageNet and Places, the MI constraints prevented the performance degradation seen in naive combinations, achieving higher accuracy on both datasets' validation sets.
- Enhanced Transferability: Models pre-trained on ImageNet + Places using this method showed superior performance when transferred to PASCAL VOC classification tasks, proving that multi-domain pre-training creates more versatile features.
Limitations and Future Work
The model's performance is sensitive to the hyperparameters ($\lambda_u$ and $\lambda_l$) used to balance the MI constraints, requiring careful tuning. Additionally, the approach currently assumes domains are clearly labeled. Future work could explore "latent" domains where style differences are not explicitly labeled but must be discovered by the model.
Summary updated