Abstract
Few-shot generative adaptation aims to fine-tune large pre-trained models on extremely limited target data. However, the inherent "domain gap" between source and target distributions often leads to structural distortions or a loss of diversity. We present Equivariant Feature Rotation (EFR), a method that aligns these distributions within a proxy feature space using learnable rotations in a parameterized Lie Group. By preserving geometric structure through equivariant transformations rather than forced alignment, EFR achieves state-of-the-art results in generating high-quality, diverse images from as few as 10 samples.
Introduction
Deep generative models, particularly Generative Adversarial Networks (GANs), have demonstrated remarkable success in synthesizing high-resolution images. However, their performance is traditionally predicated on the availability of massive datasets. Few-shot generative adaptation seeks to mitigate this data dependency by leveraging a model pre-trained on a "source" domain (e.g., human faces) and adapting it to a "target" domain (e.g., sketches or animal faces) using only a handful of examples.
The primary obstacle in this task is the domain gap. When the target distribution is forced to align too rigidly with the source, the resulting images often suffer from artifacts or structural distortions. Conversely, insufficient alignment leads to "mode collapse," where the model fails to capture the diversity of the target domain. This paper argues that these issues arise because the feature distributions of the two domains are misaligned in orientation—essentially "pointing in different directions." To address this, the authors propose Equivariant Feature Rotation (EFR), which utilizes learnable rotations to bridge the domain gap without compromising structural integrity.
Methodology: Equivariant Feature Rotation
The core innovation of EFR is the transition from direct feature alignment to alignment within a "proxy feature space" via rotation.
Equivariant Self-Rotation
Rather than "stretching" or "squishing" features to match, EFR learns a rotation matrix ($R^*$) within a parameterized Lie Group $SO(d)$. This rotation is equivariant, meaning it alters the orientation of the feature distribution while preserving its internal geometric relationships. This allows the target domain to "turn" toward the source domain, facilitating a more natural alignment that respects the underlying structure learned during pre-training.
Optimization in Lie Algebra
To maintain the validity of the rotation matrix (ensuring it remains orthogonal) during standard gradient descent, the authors optimize within the Lie Algebra $so(d)$ using skew-symmetric matrices. This mathematical framework ensures that the transformation remains a pure rotation, avoiding the distortions common in unconstrained optimization.
Two-Level Alignment Strategy
EFR employs a dual-level optimization process to ensure both local consistency and global diversity:
- Instance-wise Alignment: A contrastive loss is utilized to ensure that specific latent codes produce smooth, meaningful transitions from source content to target style.
- Distribution-level Matching: To capture the global structure, the method employs Optimal Transport (OT) theory, specifically the Gromov-Wasserstein distance. Unlike standard distance metrics, Gromov-Wasserstein measures how the relationships between points in the source domain compare to the relationships in the target domain. This relational alignment is crucial for maintaining diversity when training data is extremely sparse.
Experimental Results
The effectiveness of EFR was evaluated across several benchmarks, including FFHQ (faces), LSUN Churches, and LSUN Cars, adapted to target domains such as MetFaces, AFHQ-Cat, and various artistic sketches.
Quantitative Performance
EFR consistently established new state-of-the-art (SOTA) benchmarks across multiple backbones, including StyleGAN2 and Diffusion Models. Notably, in the FFHQ → AFHQ-Cat (10-shot) task, EFR achieved a Frechet Inception Distance (FID) of 53.91, a significant improvement over previous methods like RSSA and CDC, which typically scored above 58. Furthermore, higher intra-LPIPS scores confirmed that EFR maintains superior image diversity, avoiding the common pitfall of generating repetitive samples.
Qualitative Analysis
Visually, EFR avoids the "hybridization" artifacts seen in earlier methods. For instance, when adapting from human faces to cats, previous models often produced feline features with unsettlingly human-like proportions. EFR successfully generates distinct, realistic feline features while preserving the pose, lighting, and composition of the source images.
Discussion and Conclusion
The success of EFR highlights a fundamental insight: rotation is a more structure-preserving mechanism for domain alignment than traditional feature warping. By performing alignment in a self-rotated proxy space, the model avoids the entanglement of attributes that typically causes artifacts.
While the method introduces some computational complexity through the use of Gromov-Wasserstein distance and Lie Group operations, the gains in image quality and diversity are substantial. Future work may explore the application of equivariant rotations to the cross-attention layers of large-scale text-to-image diffusion models, potentially extending these benefits to a broader range of generative architectures.
Summary updated