Introduction
Generative Adversarial Networks (GANs) like StyleGAN and BigGAN have achieved remarkable success in synthesizing high-fidelity images. However, a persistent challenge remains: achieving precise spatial control over the generated content. Because GAN latent codes are typically global, modifying a single attribute (such as hair length) often results in unintended changes to unrelated regions (such as eye color or background). Researchers have introduced Inference-Time Adaptive Normalization (ITAN), a "plug-and-play" technique that enables localized edits without requiring additional training or fine-tuning.
The Problem: Global Latent Codes
Standard GAN architectures apply latent codes globally across the entire image. When a latent vector is transformed into normalization parameters—specifically "gain" ($\gamma$) and "bias" ($\beta$)—these values scale and shift feature maps uniformly. This entanglement makes it difficult to isolate specific regions for editing. Existing solutions often involve training auxiliary masking networks or performing computationally expensive fine-tuning, which limits their versatility across different model architectures.
Methodology: Hijacking Normalization
ITAN addresses this by modifying the normalization layers of pre-trained models at inference time. The core innovation lies in making the normalization step spatially aware. Instead of applying a single set of parameters globally, the method allows for the blending of multiple latent codes based on a user-provided spatial mask.
The ITAN Process:
- Spatial Masking: A user provides a mask $m$ to define the region of interest.
- Parameter Blending: The system calculates different sets of $\gamma$ and $\beta$ from distinct latent codes and blends them spatially (e.g., $\hat{\gamma} = m \cdot \gamma_1 + (1-m) \cdot \gamma_2$).
- Zero-Shot Execution: This process occurs entirely during the forward pass. No weights in the pre-trained GAN are altered, preserving the original model's visual quality.
Because most state-of-the-art GANs utilize adaptive normalization (such as AdaIN in StyleGAN or Batch-Norm in BigGAN), ITAN is compatible with a wide range of modern architectures.
Key Applications and Results
The researchers validated ITAN using StyleGAN2 (trained on the FFHQ face dataset) and BigGAN (trained on ImageNet). The method demonstrated several powerful capabilities:
- Local Generation: Resampling specific regions (e.g., changing only the bottom half of a face) while keeping the rest of the image identical.
- Attribute Transfer: Selectively applying attributes like "glasses" from one latent code to a specific region of another image.
- Class Hybridization: Creating "chimera" objects by using different class embeddings for different parts of an image, such as placing a cat's features onto a dog's body.
By operating in the feature space of the generator's internal layers, ITAN naturally produces smooth transitions between regions, even when using coarse, hand-drawn masks.
Limitations and Future Work
While ITAN offers a significant speedup over fine-tuning methods, it currently relies on external spatial masks provided by the user or segmentation tools. Additionally, editing real-world photographs requires first "inverting" the image into the GAN's latent space, which can introduce artifacts. Future research may focus on automating mask generation and improving the fidelity of real-image inversion to broaden the practical utility of spatial normalization control.
Summary updated