Cross-Domain Latent Modulation for Variational Transfer Learning

Jinyong Hou, Jeremiah D. Deng, Stephen Cranefield, Xuejie DingWACV 2021

TL;DR

This paper introduces Cross-Domain Latent Modulation (CDLM), a novel variational autoencoder (VAE) framework that enhances transfer learning by modulating latent representations with deep features from a different domain. CDLM achieves competitive performance in unsupervised domain adaptation and image-to-image translation tasks by effectively bridging domain gaps.

What Problem Does This Paper Address?

Transfer learning is crucial for applying models trained on one dataset or task to another, as performance often degrades significantly due to domain shifts or sampling biases. A key challenge in cross-domain transfer is learning a joint distribution across domains, which is notoriously ill-posed. Existing methods often assume a shared latent space or use adversarial strategies to align latent representations. However, these approaches sometimes overlook the generative aspect of latent spaces, which could be leveraged for better cross-domain transfer. This paper aims to address this by proposing a method that explicitly uses deep representations from one domain to modulate the latent space of another, facilitating more effective transfer.

Methods

The core of CDLM lies in its novel "transfer latent space" and a modified reparameterization trick.

  • Transfer Latent Space: This concept aims to create a "mixer" for different domains. It's defined by a mapping function that extracts domain information (mean $\mu$ and standard deviation $\sigma$) and deep representations ($h$) from input data. A transfer map $G$ then generates latent variables by combining these elements from different domains.
  • Modified Reparameterization: Instead of the standard VAE reparameterization using Gaussian noise, CDLM uses a weighted sum of deep features from the other domain and standard Gaussian noise. Specifically, for a source domain latent variable $\tilde{z}_{st}$, it's generated as $\mu_s + \sigma_s \odot (\gamma_1 h_t + \gamma_2 \epsilon)$, where $h_t$ is the deep representation from the target domain, $\epsilon$ is Gaussian noise, and $\gamma_1, \gamma_2$ are hyperparameters balancing the influence of deep features and noise. This allows for cross-domain modulation of latent representations.
  • Adversarial Alignment: To further align the latent spaces of different domains at the feature level, a gradient reversal layer is employed for adversarial training, making representations more domain-invariant.
  • Consistency Constraints: The model enforces consistency between reconstructions from modulated latent encodings and generations from deep representations. This helps in achieving inter-class alignment within the latent space.
  • Unified Inference Model: A single inference model (encoder) is used for both source and target domains, simplifying the architecture.

Datasets

The paper evaluates CDLM on a variety of benchmark datasets across unsupervised domain adaptation and image-to-image translation tasks:

  • Unsupervised Domain Adaptation:
    • MNIST
    • USPS
    • MNISTM (a modified version of MNIST with random backgrounds)
    • Fashion-MNIST
    • FashionM (Fashion-MNIST with random backgrounds)
    • Linemod 3D (synthetic images)
    • Linemod Real (real images)
    • CelebA (for face sketch translation)
    • UT-Zap50K-shoes (for edge-to-shoe translation)

Key Insights and Ideas

  • Deep Features as Modulators: The most significant insight is using deep representations from one domain to directly modulate the latent space of another. This goes beyond simple latent space alignment and actively injects cross-domain information into the generative process.
  • Reparameterization as a Control Knob: The modified reparameterization trick acts as a flexible mechanism. By adjusting hyperparameters $\gamma_1$ and $\gamma_2$, one can control the extent to which deep features from the source/target domain influence the latent representation of the other.
  • Bridging Domain Gaps via Generation: The consistency constraints between reconstruction and generation are crucial for aligning latent representations and improving inter-class discrimination, effectively bridging the domain gap.
  • Simplicity and Effectiveness: CDLM achieves competitive results with a relatively straightforward architecture, demonstrating that complex disentanglement might not always be necessary for effective cross-domain transfer.

Key Results

  • Unsupervised Domain Adaptation: CDLM achieves competitive, and often state-of-the-art, classification accuracies across various domain adaptation tasks. For instance, it shows strong performance on tasks with larger domain gaps like MNIST→MNISTM and Fashion→FashionM. In the Linemod 3D scenario, it outperforms most compared methods.
  • Image-to-Image Translation: The model demonstrates effective image mapping, outperforming E-CDRD and matching StarGAN in performance metrics like SSIM, MSE, and PSNR. Qualitative results show realistic translations, such as converting edges to shoes or sketches to faces.
  • Latent Space Alignment: t-SNE visualizations confirm that CDLM effectively aligns latent encodings from different domains.
  • A-Distance Reduction: CDLM significantly reduces domain discrepancy (A-distance) compared to raw features and DANN, indicating that its generated images are harder to distinguish by domain.

Limitations and Future Work

  • Limitations:
    • The paper notes that for face-to-sketch translation, the generated sketches might lose some details, possibly because low-level features are neglected when deep features act as the condition.
  • While competitive, the performance on the Linemod 3D task is slightly lower than PixelDA, which utilizes depth images in addition to RGB.
  • Future Work: The authors intend to further improve their variational transfer learning framework and extend it to handle heterogeneous, multi-domain transfer tasks.

Summary updated

BibTeX
@inproceedings{hou2021cross,
  author = {Jinyong Hou and Jeremiah D. Deng and Stephen Cranefield and Xuejie Ding},
  title = {{Cross-Domain Latent Modulation for Variational Transfer Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure