Augmented Cyclic Adversarial Learning for Low Resource Domain Adaptation

Ehsan Hosseini-Asl, Yingbo Zhou, Caiming Xiong, Richard Socher

This paper from Salesforce Research introduces Augmented Cyclic Adversarial Learning (ACAL). It tackles a common bottleneck in machine learning: how to make a model work in a new "target" domain when you have very little labeled data for it, but plenty of data for a related "source" domain.

ACAL improves domain adaptation by replacing the strict "pixel-perfect" reconstruction loss of standard CycleGAN architectures with a "semantic" loss enforced by a task-specific model. This relaxation allows the system to focus on preserving the core meaning of data (like the digit "4" or a specific phoneme) rather than irrelevant stylistic details, leading to massive performance gains in low-resource settings.

The Bottleneck in Standard Domain Adaptation

Domain adaptation is the art of transferring knowledge from a data-rich source (e.g., synthetic images) to a data-poor target (e.g., real-world photos). A popular tool for this is CycleGAN, which learns to map data between domains using a "cycle-consistency" constraint: if you translate an image from Domain A to B and back to A, you should get the original image back.

However, the authors argue that CycleGAN’s standard reconstruction loss (usually an $L_1$ pixel-wise comparison) is too restrictive. In low-resource scenarios, the model wastes its limited capacity trying to reconstruct every noisy pixel or background detail to satisfy the $L_1$ loss. This often prevents the model from learning the actual underlying task, especially when the "discriminator" (the part of the GAN that tells real data from fake) overfits due to the lack of target samples.

Proposed Method: Relaxing and Augmenting Cycle-Consistency

The authors propose two major modifications to the standard CycleGAN framework to alleviate these bottlenecks:

Comparison of CycleGAN, Relaxed, and Augmented Cycle-Consistency
  1. Relaxed Cycle-Consistency (RCAL): Instead of requiring the reconstructed image to look exactly like the original pixel-for-pixel ($x \approx x'$), they use a pre-trained task-specific model (like a digit classifier or speech recognizer) to ensure the meaning is preserved. If the original image was a "7," the round-trip version just needs to be recognized as a "7" by the task model. This replaces the rigid $L_1$ loss with a semantic $\mathcal{L}_{task}$ loss.
  2. Augmented Discriminator (ACAL): They further "augment" the learning process by using the task-specific model to help the discriminator. The task model provides a conditional probability signal that acts as an additional source of information, helping the generator learn the target distribution even when the primary discriminator is struggling with limited data.

The paper provides algorithms for both supervised (where a few target labels exist) and unsupervised (where no target labels exist, and the source model is used to estimate them) settings.

Key Results and Insights

The researchers evaluated ACAL across two distinct modalities: Computer Vision (digits like MNIST, SVHN, USPS) and Speech Recognition (the TIMIT dataset, adapting between male and female speakers).

The performance jumps were significant, particularly in the most difficult "low-resource" scenarios:

Accuracy comparison in low-resource settings
  • Massive Gains over CycleGAN: On the SVHN to MNIST task using only 10 labeled samples per class, standard CycleGAN achieved only 45.5% accuracy, while ACAL reached an impressive 93.9%.
  • Supervised Low-Resource Adaptation: In a supervised setting with very few samples, ACAL improved absolute accuracy by 14% over standard baselines when adapting SVHN to MNIST.
  • Speech Adaptation: On the TIMIT dataset, treating male speakers as the high-resource source and female speakers as the low-resource target, ACAL reduced the Phoneme Error Rate (PER) for female speakers by approximately 2% absolute. This is a significant margin in speech tasks, allowing the model to nearly match the performance of a model trained on the full female speaker dataset.

Core Takeaways: * Semantic > Structural: In many tasks, we don't care if a translated image has the exact same background noise as the source; we only care that the object of interest is preserved. By relaxing the cycle constraint, the model gains the "freedom" to adapt style more aggressively. * The Discriminator Bottleneck: In low-resource settings, the GAN discriminator often becomes a "delta function"—it simply memorizes the few available target samples. ACAL bypasses this by using the task model as a more stable "teacher." * Double Cycles Matter: Having cycles running in both directions (Source $\to$ Target $\to$ Source AND Target $\to$ Source $\to$ Target) is essential for stability when one domain is much smaller than the other.

Limitations and Future Work

  • Dependency on Task Models: ACAL requires a task-specific model (like a classifier) to be available. While common for digits or speech, this might be harder to implement for abstract generative tasks where a clear "task model" doesn't exist.
  • Architecture Sensitivity: The complexity of the task model matters. Using a very deep model (like DenseNet) as the task-enforcer on a simple dataset (like MNIST) can actually lead to overfitting and worse gradients for the generator.
  • Future Directions: The authors suggest extending this to unsupervised tasks like language modeling by using self-supervised models (like autoencoders or Transformers) as the "task" enforcers.

Summary updated

Abstract

Training a model to perform a task typically requires a large amount of data from the domains in which the task will be applied. However, it is often the case that data are abundant in some domains but scarce in others. Domain adaptation deals with the challenge of adapting a model trained from a data-rich source domain to perform well in a data-poor target domain. In general, this requires learning plausible mappings between domains. CycleGAN is a powerful framework that efficiently learns to map inputs from one domain to another using adversarial training and a cycle-consistency constraint. However, the conventional approach of enforcing cycle-consistency via reconstruction may be overly restrictive in cases where one or more domains have limited training data. In this paper, we propose an augmented cyclic adversarial learning model that enforces the cycle-consistency constraint via an external task specific model, which encourages the preservation of task-relevant content as opposed to exact reconstruction. We explore digit classification in a low-resource setting in supervised, semi and unsupervised situation, as well as high resource unsupervised. In low-resource supervised setting, the results show that our approach improves absolute performance by 14% and 4% when adapting SVHN to MNIST and vice versa, respectively, which outperforms unsupervised domain adaptation methods that require high-resource unlabeled target domain. Moreover, using only few unsupervised target data, our approach can still outperforms many high-resource unsupervised models. Our model also outperforms on USPS to MNIST and synthetic digit to SVHN for high resource unsupervised adaptation. In speech domains, we similarly adopt a speech recognition model from each domain as the task specific model. Our approach improves absolute performance of speech recognition by 2% for female speakers in the TIMIT dataset, where the majority of training samples are from male voices.
BibTeX
@inproceedings{hosseiniasl2019augmented,
  author = {Ehsan Hosseini-Asl and Yingbo Zhou and Caiming Xiong and Richard Socher},
  title = {{Augmented Cyclic Adversarial Learning for Low Resource Domain Adaptation}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=frOs9qRHtg},
}
Figure