Self-Supervised Representation Learning From Multi-Domain Data

Zeyu Feng, Chang Xu, Dacheng Tao

Summary

This research introduces a framework for self-supervised learning (SSL) that effectively leverages data from multiple related domains, such as photos, sketches, and cartoons. By employing mutual information constraints to balance domain-invariant features with domain-specific details, the authors produce robust, unbiased representations that outperform models trained on single datasets or naively combined data.

The Problem: Dataset Bias and the Domain Gap

Most SSL methods are trained on single, large-scale datasets like ImageNet. However, these datasets are inherently biased, reflecting specific lighting, camera angles, and styles that do not represent the full "visual world."

Simply combining multiple datasets (a strategy known as "DeepAll") often leads to a drop in performance. This occurs because the model becomes confused by the "domain gap"—the stylistic differences between, for instance, a photo of a dog and a sketch of a dog. The core challenge lies in learning the high-level concept of an object without being distracted by the stylistic noise of its source domain.

Methodology: The Push-Pull of Information

The authors propose a multi-domain learning (MDL) approach governed by two competing information-theoretic constraints:

  1. Inter-domain MI Minimization (Invariance): The model is encouraged to minimize the Mutual Information (MI) between the learned representation ($z$) and the domain label ($d$). This forces the network to discard "unwanted variations" like style or lighting, making the representation invariant across domains. This is implemented via adversarial training using a Gradient Reversal Layer (GRL).
  2. Intra-domain MI Maximization (Preservation): To prevent the model from losing useful features by becoming too invariant, the authors maximize the MI between the input image ($x$) and its representation ($z$) for each domain. This ensures the model captures essential semantic details. This is implemented using the Jensen-Shannon Divergence (JSD) as a tractable lower bound of MI.

The final objective is a minimax game where the encoder satisfies the SSL task (e.g., predicting image rotation) while balancing these two constraints.

Datasets and Evaluation

The method was evaluated on three distinct setups:

  • PACS: A standard multi-domain benchmark containing four domains: Photo, Art painting, Cartoon, and Sketch. It features 7 shared categories across very different styles.
  • ImageNet & Places: Two massive datasets treated as separate domains (object-oriented vs. scene-oriented) to test scalability.
  • PASCAL VOC 2007: Used as a downstream "target" dataset to test how well the multi-domain representations transfer to new tasks.
The PACS dataset illustrates the challenge of learning consistent representations across diverse domains like Art, Cartoon, Sketch, and Photo.

Key Results and Insights

  • Naive Mixing is Suboptimal: Simply concatenating datasets often leads to worse performance than single-domain training because the model overfits to domain-specific biases.
  • PACS Performance: On the PACS dataset, the proposed method (DeepAll+MI) improved average linear classification accuracy by 1.1% on conv5 and 3.0% on fc7 layers compared to the naive DeepAll approach.
  • Large-Scale Gains: When combining ImageNet and Places, the MI constraints prevented the performance degradation seen in naive combinations, achieving higher accuracy on both datasets' validation sets.
  • Enhanced Transferability: Models pre-trained on ImageNet + Places using this method showed superior performance when transferred to PASCAL VOC classification tasks, proving that multi-domain pre-training creates more versatile features.

Limitations and Future Work

The model's performance is sensitive to the hyperparameters ($\lambda_u$ and $\lambda_l$) used to balance the MI constraints, requiring careful tuning. Additionally, the approach currently assumes domains are clearly labeled. Future work could explore "latent" domains where style differences are not explicitly labeled but must be discovered by the model.

Summary updated

Abstract

We present an information-theoretically motivated constraint for self-supervised representation learning from multiple related domains. In contrast to previous self-supervised learning methods, our approach learns from multiple domains, which has the benefit of decreasing the build-in bias of individual domain, as well as leveraging information and allowing knowledge transfer across multiple domains. The proposed mutual information constraints encourage neural network to extract common invariant information across domains and to preserve peculiar information of each domain simultaneously. We adopt tractable upper and lower bounds of mutual information to make the proposed constraints solvable. The learned representation is more unbiased and robust toward the input images. Extensive experimental results on both multi-domain and large-scale datasets demonstrate the necessity and advantage of multi-domain self-supervised learning with mutual information constraints. Representations learned in our framework on state-of-the-art methods achieve improved performance than those learned on a single domain.
BibTeX
@inproceedings{feng2019self,
  author = {Zeyu Feng and Chang Xu and Dacheng Tao},
  title = {{Self-Supervised Representation Learning From Multi-Domain Data}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00334},
}
Figure