Maximal Domain Independent Representations Improve Transfer Learning

Adrian Shuai Li, Elisa Bertino, Xuan-Hong Dang, Ankush Singla, Yuhai Tu, Mark N. WegmanarXiv 2023 · DOI · arXiv

Abstract

Domain Adaptation (DA) aims to transfer knowledge from a labeled source domain to an unlabeled or sparsely labeled target domain. A common strategy is to decompose data into domain-independent (DIRep) and domain-dependent (DDRep) representations. However, existing methods often suffer from information leakage, where label-relevant information is sequestered in the DDRep, hindering the classifier. This paper introduces MaxDIRep, a novel framework that enforces a strict information-minimization constraint on the DDRep using Kullback-Leibler (KL) divergence. This "information starvation" forces the DIRep to capture all transferable, label-relevant features. Evaluated across diverse benchmarks, including image classification and tabular network intrusion detection, MaxDIRep demonstrates superior performance and robustness against spurious correlations.

Introduction

In machine learning, models often fail when the test data distribution differs from the training data—a phenomenon known as domain shift. Domain Adaptation (DA) addresses this by aligning features between a "source" domain and a "target" domain. A significant challenge is the presence of spurious correlations, where a model learns domain-specific cues (like background color) instead of the actual object features. While methods like Domain-Separation Networks (DSN) attempt to partition features into domain-independent and domain-dependent components, they often use weak constraints that allow useful information to "leak" into the background representation.

Methods

MaxDIRep employs a five-network architecture consisting of a Generator (G), a Discriminator (D), a Classifier (C), an Encoder (E), and a Decoder (F).

The MaxDIRep architecture featuring five interconnected networks to partition domain-independent and domain-dependent representations.

The core innovation is the application of a Kullback-Leibler (KL) divergence constraint to the DDRep. By minimizing the information content of the domain-specific part, the model is forced to pack all label-relevant information into the DIRep. An adversarial discriminator ensures the DIRep remains domain-invariant, while a reconstruction loss (via the decoder) ensures that the combination of DIRep and DDRep preserves the original input's integrity.

Key Insights

The authors demonstrate that standard orthogonality constraints are insufficient for optimal domain separation. Geometrically, these constraints allow for solutions where the model ignores critical target data. By "starving" the DDRep of information, MaxDIRep pushes the representation toward a unique solution that maximizes the utility of the transferable features.

The effectiveness of this separation is visually verifiable through "bit flipping." In a simplified model where the DDRep is a single bit representing the domain, flipping this bit during reconstruction transforms a source image into the target domain's style while preserving the object's identity.

Visualizing domain separation: flipping the domain bit ($d$) during reconstruction transforms images between source and target styles.

Key Results

MaxDIRep was tested on benchmarks including Synthetic Fashion-MNIST, Office-Home, and the tabular UNSW-NB15 dataset.

  • Superiority over DSN: On the Fashion-MNIST "random cheating" task, MaxDIRep achieved 61.6% accuracy, significantly outperforming DSN (57.1%) and the popular DANN (53.6%).
  • Benchmark Performance: On the challenging Office-Home dataset, MaxDIRep reached an average accuracy of 65.6%, beating out other state-of-the-art methods like CDAN (63.8%) and GPDA (60.2%).
  • Few-Shot Success: In network intrusion detection, MaxDIRep showed the steepest performance gains as small amounts of labeled target data were introduced, proving it leverages limited information more efficiently than its predecessors.

Limitations and Future Work

While MaxDIRep performs exceptionally well, the authors note a few areas for growth. Some competing methods use conditional discriminators that account for label distributions; integrating this could potentially boost MaxDIRep’s performance. Additionally, the current version doesn't use "pseudo-labels" to refine training. Finally, the five-network architecture is more complex to train than simpler adversarial methods, requiring careful hyperparameter tuning for the various loss functions.

Summary updated

BibTeX
@misc{li2023maximal,
  author = {Adrian Shuai Li and Elisa Bertino and Xuan-Hong Dang and Ankush Singla and Yuhai Tu and Mark N. Wegman},
  title = {{Maximal Domain Independent Representations Improve Transfer Learning}},
  year = {2023},
  eprint = {2306.00262},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.00262},
}
Figure