Abstract
We consider the novel task of learning disentangled representations of object shape and appearance across multiple domains (e.g., dogs and cars). The goal is to learn a generative model that learns an intermediate distribution, which borrows a subset of properties from each domain, enabling the generation of images that did not exist in any domain exclusively. This challenging problem requires an accurate disentanglement of object shape, appearance, and background from each domain, so that the appearance and shape factors from the two domains can be interchanged. We augment an existing approach that can disentangle factors within a single domain but struggles to do so across domains. Our key technical contribution is to represent object appearance with a differentiable histogram of visual features, and to optimize the generator so that two images with the same latent appearance factor but different latent shape factors produce similar histograms. On multiple multi-domain datasets, we demonstrate our method leads to accurate and consistent appearance and shape transfer across domains.
Introduction
Generative models have achieved significant success in disentangling factors of variation within a single domain—for instance, combining the shape of one bird species with the color of another. However, inter-domain disentanglement remains a formidable challenge. This task involves combining factors from entirely different distributions, such as the shape of a car and the appearance of a dog.
The primary obstacle is the absence of ground-truth data for hybrid concepts like a "furry car." Standard Generative Adversarial Networks (GANs) fail because discriminators penalize out-of-distribution (OOD) hybrids as "fake." Furthermore, existing image-to-image translation methods often rely on structural correspondences that do not exist between disparate categories like birds and cars.
Methodology
To address these challenges, we extend the FineGAN framework, which disentangles background, shape, and appearance within single domains. Our approach introduces three key innovations to enable cross-domain transfer:
- Differentiable Feature Histograms: We represent an object's appearance using the frequency of low-level visual concepts (colors and textures). This differentiable histogram serves as a heuristic to match appearance without requiring a discriminator for hybrid images.
- Contrastive Filter Learning: Instead of hand-crafted filters, we train a filter bank using the NT-Xent contrastive loss. This allows the model to learn data-driven, domain-specific features such as leopard spots or bird feathers.
- Histogram-Based Optimization: The generator is optimized to ensure that images sharing the same latent appearance code produce similar feature histograms, regardless of their shape or domain. Crucially, adversarial loss is disabled for inter-domain hybrids to prevent the discriminator from suppressing novel combinations.
Experimental Setup
We evaluated the model across several multi-domain pairings using the following datasets:
- CUB: 200 bird categories.
- Stanford Dogs: 120 dog categories.
- Stanford Cars: 196 car categories.
- Animal Faces: 149 carnivorous animal categories.
Test settings included birds $\leftrightarrow$ cars, cars $\leftrightarrow$ dogs, dogs $\leftrightarrow$ birds, and animals $\leftrightarrow$ animals.
Results and Discussion
Our method demonstrates superior performance in both appearance retention and shape preservation.
Quantitative Performance
- Appearance Transfer: In the birds $\leftrightarrow$ cars setting, our method achieved a $\chi^2$-distance for color histograms of 0.2484, significantly lower than FineGAN (0.4295) and AdaIN (0.5619).
- Shape Retention: We measured shape disentanglement using Intersection over Union (IoU) on foreground masks. Our method achieved a score of 0.861 in the birds $\leftrightarrow$ cars task, outperforming the baseline (0.831).
- Human Evaluation: In perceptual studies on Amazon Mechanical Turk, users preferred our hybrid images over AdaIN in 81.39% of cases and over FineGAN in 74.38% of cases.
Key Insights
- Frequency Matching as a Bridge: Statistical matching of low-level features is a robust proxy for appearance transfer when adversarial training is infeasible.
- Isolating Appearance from Style: By masking backgrounds and focusing on learned texture histograms, the model successfully separates object-specific appearance from global image style.
Conclusion
The framework currently assumes a specific data hierarchy where shape variation is less than appearance variation. Additionally, training complexity scales quadratically with the number of domains. Future work will explore more efficient scaling and applications in AI-assisted design, such as visualizing furniture with arbitrary animal textures. This cross-domain capability serves as a rigorous new benchmark for evaluating true feature disentanglement in generative models.
Summary updated