A Turn Toward Better Alignment: Few-Shot Generative Adaptation with Equivariant Feature Rotation

Chenghao Xu, Qi Liu, Jiexi Yan, Muli Yang, Cheng DengarXiv 2025 · DOI · arXiv

Abstract

Few-shot generative adaptation aims to fine-tune large pre-trained models on extremely limited target data. However, the inherent "domain gap" between source and target distributions often leads to structural distortions or a loss of diversity. We present Equivariant Feature Rotation (EFR), a method that aligns these distributions within a proxy feature space using learnable rotations in a parameterized Lie Group. By preserving geometric structure through equivariant transformations rather than forced alignment, EFR achieves state-of-the-art results in generating high-quality, diverse images from as few as 10 samples.

Introduction

Deep generative models, particularly Generative Adversarial Networks (GANs), have demonstrated remarkable success in synthesizing high-resolution images. However, their performance is traditionally predicated on the availability of massive datasets. Few-shot generative adaptation seeks to mitigate this data dependency by leveraging a model pre-trained on a "source" domain (e.g., human faces) and adapting it to a "target" domain (e.g., sketches or animal faces) using only a handful of examples.

The primary obstacle in this task is the domain gap. When the target distribution is forced to align too rigidly with the source, the resulting images often suffer from artifacts or structural distortions. Conversely, insufficient alignment leads to "mode collapse," where the model fails to capture the diversity of the target domain. This paper argues that these issues arise because the feature distributions of the two domains are misaligned in orientation—essentially "pointing in different directions." To address this, the authors propose Equivariant Feature Rotation (EFR), which utilizes learnable rotations to bridge the domain gap without compromising structural integrity.

Methodology: Equivariant Feature Rotation

The core innovation of EFR is the transition from direct feature alignment to alignment within a "proxy feature space" via rotation.

Equivariant Self-Rotation

Rather than "stretching" or "squishing" features to match, EFR learns a rotation matrix ($R^*$) within a parameterized Lie Group $SO(d)$. This rotation is equivariant, meaning it alters the orientation of the feature distribution while preserving its internal geometric relationships. This allows the target domain to "turn" toward the source domain, facilitating a more natural alignment that respects the underlying structure learned during pre-training.

Optimization in Lie Algebra

To maintain the validity of the rotation matrix (ensuring it remains orthogonal) during standard gradient descent, the authors optimize within the Lie Algebra $so(d)$ using skew-symmetric matrices. This mathematical framework ensures that the transformation remains a pure rotation, avoiding the distortions common in unconstrained optimization.

Two-Level Alignment Strategy

EFR employs a dual-level optimization process to ensure both local consistency and global diversity:

  1. Instance-wise Alignment: A contrastive loss is utilized to ensure that specific latent codes produce smooth, meaningful transitions from source content to target style.
  2. Distribution-level Matching: To capture the global structure, the method employs Optimal Transport (OT) theory, specifically the Gromov-Wasserstein distance. Unlike standard distance metrics, Gromov-Wasserstein measures how the relationships between points in the source domain compare to the relationships in the target domain. This relational alignment is crucial for maintaining diversity when training data is extremely sparse.

Experimental Results

The effectiveness of EFR was evaluated across several benchmarks, including FFHQ (faces), LSUN Churches, and LSUN Cars, adapted to target domains such as MetFaces, AFHQ-Cat, and various artistic sketches.

Quantitative Performance

EFR consistently established new state-of-the-art (SOTA) benchmarks across multiple backbones, including StyleGAN2 and Diffusion Models. Notably, in the FFHQ → AFHQ-Cat (10-shot) task, EFR achieved a Frechet Inception Distance (FID) of 53.91, a significant improvement over previous methods like RSSA and CDC, which typically scored above 58. Furthermore, higher intra-LPIPS scores confirmed that EFR maintains superior image diversity, avoiding the common pitfall of generating repetitive samples.

Qualitative Analysis

Visually, EFR avoids the "hybridization" artifacts seen in earlier methods. For instance, when adapting from human faces to cats, previous models often produced feline features with unsettlingly human-like proportions. EFR successfully generates distinct, realistic feline features while preserving the pose, lighting, and composition of the source images.

Discussion and Conclusion

The success of EFR highlights a fundamental insight: rotation is a more structure-preserving mechanism for domain alignment than traditional feature warping. By performing alignment in a self-rotated proxy space, the model avoids the entanglement of attributes that typically causes artifacts.

While the method introduces some computational complexity through the use of Gromov-Wasserstein distance and Lie Group operations, the gains in image quality and diversity are substantial. Future work may explore the application of equivariant rotations to the cross-attention layers of large-scale text-to-image diffusion models, potentially extending these benefits to a broader range of generative architectures.

Summary updated

Abstract

Few-shot image generation aims to effectively adapt a source generative model to a target domain using very few training images. Most existing approaches introduce consistency constraints-typically through instance-level or distribution-level loss functions-to directly align the distribution patterns of source and target domains within their respective latent spaces. However, these strategies often fall short: overly strict constraints can amplify the negative effects of the domain gap, leading to distorted or uninformative content, while overly relaxed constraints may fail to leverage the source domain effectively. This limitation primarily stems from the inherent discrepancy in the underlying distribution structures of the source and target domains. The scarcity of target samples further compounds this issue by hindering accurate estimation of the target domain's distribution. To overcome these limitations, we propose Equivariant Feature Rotation (EFR), a novel adaptation strategy that aligns source and target domains at two complementary levels within a self-rotated proxy feature space. Specifically, we perform adaptive rotations within a parameterized Lie Group to transform both source and target features into an equivariant proxy space, where alignment is conducted. These learnable rotation matrices serve to bridge the domain gap by preserving intra-domain structural information without distortion, while the alignment optimization facilitates effective knowledge transfer from the source to the target domain. Comprehensive experiments on a variety of commonly used datasets demonstrate that our method significantly enhances the generative performance within the targeted domain.
BibTeX
@misc{xu2025turn,
  author = {Chenghao Xu and Qi Liu and Jiexi Yan and Muli Yang and Cheng Deng},
  title = {{A Turn Toward Better Alignment: Few-Shot Generative Adaptation with Equivariant Feature Rotation}},
  year = {2025},
  eprint = {2512.21174},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.21174},
}
Figure