Bidirectional One-Shot Unsupervised Domain Mapping

Tomer Cohen, Lior WolfICCV 2019 · OpenReview

Abstract

We study the problem of mapping between a domain $A$, in which there is a single training sample and a domain $B$, for which we have a richer training set. The method we present is able to perform this mapping in both directions. For example, we can transfer all MNIST images to the visual domain captured by a single SVHN image and transform the SVHN image to the domain of the MNIST images. Our method is based on employing one encoder and one decoder for each domain, without utilizing weight sharing. The autoencoder of the single sample domain is trained to match both this sample and the latent space of domain $B$. Our results demonstrate convincing mapping between domains, where either the source or the target domain are defined by a single sample, far surpassing existing solutions.

Introduction

Unsupervised domain mapping involves learning to translate images from one visual style (Domain A) to another (Domain B) without paired training data. While standard approaches require large datasets for both domains, the "one-shot" scenario—where one domain is represented by a single image—presents a significant challenge. Existing methods often struggle with bidirectionality, particularly when mapping from a rich domain back to a one-shot domain, as training a GAN discriminator on a single image is impractical.

This paper introduces BiOST (Bidirectional One-Shot Translation), a framework that achieves state-of-the-art results in bidirectional mapping by moving away from Generative Adversarial Networks (GANs) and weight sharing. Instead, it utilizes a novel feature cycle loss and independent autoencoders to maintain content integrity and visual style.

Methodology

BiOST departs from standard literature in two primary ways: it avoids GANs in favor of a Variational Autoencoder (VAE) approach and rejects weight sharing between domain-specific encoders and decoders.

Architecture and Training

The training process occurs in two phases:

  1. Phase I: An autoencoder for the rich Domain B is trained. Its latent space is constrained to a Gaussian distribution using a variational loss, establishing a structured latent prior.
  2. Phase II: Domain A’s autoencoder is initialized as a clone of B’s. The two are then trained jointly.

To align the domains without weight sharing, the authors introduce a Feature Cycle Loss ($L_{f-cycle}$). This loss ensures that a feature from Domain B, when decoded into Domain A and re-encoded, returns to its original position in the latent space. Selective backpropagation is used to ensure Domain A adapts to Domain B's latent space.

Architecture of the BiOST method illustrating the three cycle losses used for training.

Experiments and Results

The model was evaluated on benchmarks including digits (MNIST/SVHN), faces (CelebA), and landscapes.

Key Insights

  • Weight Sharing: Contrary to many-shot models like UNIT, weight sharing was found to be detrimental in the one-shot context, as it overly restricts the decoders and leads to blurriness.
  • Latent Consistency: Enforcing consistency in the latent (feature) space is more effective than pixel-level consistency for maintaining content when data is extremely scarce.

Quantitative and Qualitative Performance

BiOST significantly outperforms the previous state-of-the-art, One-Shot Translation (OST). In MNIST to SVHN tasks, BiOST achieved 66.5% accuracy compared to OST’s 23.5%. For face hair color translation, BiOST achieved a content retention score of 0.53 (via VGG descriptors), surpassing OST's 0.39.

Qualitative comparison between the baseline OST and BiOST (Ours) for hair color translation on the CelebA dataset.

Qualitatively, BiOST produces sharper details and fewer artifacts in landscape tasks compared to CycleGAN or MUNIT, which often fail when restricted to a single target image.

Conclusion

BiOST demonstrates that bidirectional one-shot domain mapping is feasible without GANs or weight sharing. While the model's understanding of the one-shot domain is inherently limited by the single sample provided, it paves the way for more "ecological" AI systems capable of learning from minimal real-world examples. Future work may address resolution constraints and the impact of outlier samples in the one-shot domain.

Summary updated

Abstract

We study the problem of mapping between a domain A, in which there is a single training sample and a domain B, for which we have a richer training set. The method we present is able to perform this mapping in both directions. For example, we can transfer all MNIST images to the visual domain captured by a single SVHN image and transform the SVHN image to the domain of the MNIST images. Our method is based on employing one encoder and one decoder for each domain, without utilizing weight sharing. The autoencoder of the single sample domain is trained to match both this sample and the latent space of domain B. Our results demonstrate convincing mapping between domains, where either the source or the target domain are defined by a single sample, far surpassing existing solutions. Our code is made publicly available at https://github.com/tomercohen11/BiOST.
BibTeX
@inproceedings{cohen2019bidirectional,
  author = {Tomer Cohen and Lior Wolf},
  title = {{Bidirectional One-Shot Unsupervised Domain Mapping}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=6JejCLoZOfI},
}
Figure