Focal Frequency Loss for Generative Models

Liming Jiang, Bo Dai, Wayne Wu, Chen Change Loy

Introduction

Generative models such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) have achieved remarkable success in image synthesis. However, a persistent challenge remains: the inability to faithfully reproduce high-frequency details, often resulting in blurry outputs or unnatural periodic artifacts. This discrepancy is deeply rooted in the "spectral bias" of neural networks, where models naturally prioritize learning low-frequency components (coarse structures) over high-frequency ones (fine textures). Standard spatial losses, such as $L_1$ or $L_2$ pixel distances, fail to address this bias as they treat all pixels with uniform importance.

Comparison of real and generated images across different models, highlighting the discrepancy in their frequency spectra.

Methodology

To bridge this gap, we introduce Focal Frequency Loss (FFL). The core idea is to optimize the model directly in the frequency domain. The process involves three key components:

  1. Frequency Representation: Utilizing the Discrete Fourier Transform (DFT), images are mapped to the frequency domain. Crucially, FFL accounts for both amplitude and phase information, as neglecting either leads to significant reconstruction errors (e.g., irregular color patterns or excessive noise).
  2. Frequency Distance: Fourier coefficients are mapped to Euclidean vectors in a 2D plane (representing real and imaginary parts). The distance is calculated as the squared Euclidean distance between these vectors, ensuring differentiability for backpropagation.
  3. Dynamic Spectrum Weighting (The "Focal" Part): Inspired by Focal Loss in object detection, FFL employs a dynamic weight matrix. This matrix adaptively down-weights "easy" frequencies where the model performs well and up-weights "hard" frequencies where the gap between real and synthesized images is largest. This forces the generator to actively focus on missing high-frequency details during training.

Experiments and Results

The effectiveness of FFL was evaluated across diverse tasks, including texture synthesis (DTD), face reconstruction (CelebA/CelebA-HQ), and image-to-image translation (Cityscapes, ADE20K). Integrating FFL into existing architectures like VAE, pix2pix, and StyleGAN2 yielded significant improvements.

Qualitative comparison showing the impact of FFL on texture (DTD) and face (CelebA) synthesis. FFL significantly reduces blurriness and restores fine details.

Quantitatively, FFL consistently improved standard metrics such as Fréchet Inception Distance (FID) and Inception Score (IS). For instance, in VAE-based face reconstruction on CelebA, the FID was reduced from 97.0 to 83.8. Furthermore, the introduction of Log Frequency Distance (LFD) confirmed that FFL effectively narrows the frequency-domain gap better than baseline models. Qualitatively, the results are markedly sharper; VAEs produce clearer facial features, and GANs exhibit fewer "checkerboard" artifacts caused by upsampling layers.

Conclusion

Focal Frequency Loss provides a computationally efficient and effective complement to traditional spatial losses. By adaptively focusing on hard-to-synthesize frequency components, it enables generative models to overcome spectral bias and produce high-fidelity images with realistic fine-grained textures. Future research may explore alternative frequency representations or the development of inherently frequency-aware network architectures.

Summary updated

BibTeX
@inproceedings{jiang2021focal,
  author = {Liming Jiang and Bo Dai and Wayne Wu and Chen Change Loy},
  title = {{Focal Frequency Loss for Generative Models}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01366},
}
Figure