High-Fidelity and Arbitrary Face Editing

Yue Gao, Fangyun Wei, Jianmin Bao, Shuyang Gu, Dong Chen, Fang Wen, Zhouhui Lian

Abstract

Cycle consistency is widely used for face editing. However, we observe that the generator tends to find a tricky way to hide information from the original image to satisfy the constraint of cycle consistency, making it impossible to maintain the rich details (e.g., wrinkles and moles) of nonediting areas. In this work, we propose a simple yet effective method named HifaFace to address this problem from two perspectives. First, we relieve the pressure of the generator to synthesize rich details by directly feeding the high-frequency information of the input image into the end of the generator. Second, we adopt an additional discriminator to encourage the generator to synthesize rich details. Specifically, we apply wavelet transformation to transform the image into multi-frequency domains, among which the high-frequency parts can be used to recover the rich details. We also propose a novel attribute regression loss for fine-grained and wider-range control. Powered by this framework, we achieve high-fidelity and arbitrary face editing, outperforming other state-of-the-art approaches.

Introduction

Face editing aims to modify specific attributes—such as adding glasses or a smile—while preserving the subject's identity and the fine details of the original image. Most state-of-the-art models rely on cycle consistency: if you turn a person with no glasses into someone with glasses, and then turn them back, you should recover the original image.

However, the authors identify a "tricky" behavior: the generator often learns to perform steganography. Instead of realistically rendering the edited face, it hides the original face's fine details (like a mole or a specific wrinkle) as imperceptible high-frequency signals in the edited image. This satisfies the mathematical cycle but results in an edited image that looks blurry or loses the "soul" of the original subject, as illustrated in Figure 1.

Figure 1: Comparison of standard cycle-consistent editing vs. HifaFace. Standard methods (b, c) often hide original details as steganographic noise, while HifaFace (d) preserves high-frequency textures.

Methodology

HifaFace introduces three primary innovations to force the generator to be honest and high-fidelity:

1. Wavelet-Based Generator

Instead of standard skip connections, the model uses Haar Wavelet Transformation. It decomposes the input image into four frequency bands: one low-frequency (LL) and three high-frequency (LH, HL, HH). The high-frequency bands, which contain the "rich details," are fed directly into the end of the decoder. This relieves the generator from the "pressure" of having to reinvent these details from scratch.

2. High-Frequency Discriminator ($D_H$)

To ensure the high-frequency parts of the generated image look like real skin texture and not hidden "steganographic" noise, the authors add a second discriminator that specifically inspects the wavelet-decomposed high-frequency components.

3. Attribute Regression Loss

Most models treat attributes as binary (0 or 1). HifaFace introduces a regression loss that allows for a continuous scale ($\alpha$). This enables "arbitrary" editing—for example, making a smile wider or turning regular eyeglasses into dark sunglasses by increasing the attribute intensity beyond the training labels.

Experiments and Results

The researchers evaluated HifaFace on two major benchmarks: CelebA-HQ and FFHQ. The model significantly outperformed existing approaches like StarGAN, RelGAN, and StyleFlow across several metrics:

  • FID (Fréchet Inception Distance): Achieved a score of 4.04, significantly lower (better) than the next best competitor (SSCGAN at 4.69).
  • Accuracy: Reached 97.5% attribute classification accuracy, showing that the edits are highly effective.
  • SRE (Self-Reconstruction Error): Achieved the lowest error (0.021), proving it preserves the original image's identity better than other methods.

Qualitative results (e.g., Figure 2) demonstrate that while other models "wash out" the skin or lose hair detail, HifaFace maintains photographic sharpness even during extreme attribute changes.

Figure 2: High-fidelity face editing result demonstrating the preservation of fine-grained facial features.

Conclusion

HifaFace addresses the steganography problem in cycle-consistent GANs by separating structure and texture through frequency domains. By injecting high-frequency information directly and employing a specialized discriminator, the framework achieves high-fidelity results that maintain the "rich details" of the original subject. This approach not only improves visual quality but also allows for fine-grained, arbitrary control over facial attributes.

Summary updated

Abstract

Cycle consistency is widely used for face editing. However, we observe that the generator tends to find a tricky way to hide information from the original image to satisfy the constraint of cycle consistency, making it impossible to maintain the rich details (e.g., wrinkles and moles) of nonediting areas. In this work, we propose a simple yet effective method named HifaFace to address the above-mentioned problem from two perspectives. First, we relieve the pressure of the generator to synthesize rich details by directly feeding the high-frequency information of the input image into the end of the generator. Second, we adopt an additional discriminator to encourage the generator to synthesize rich details. Specifically, we apply wavelet transformation to transform the image into multi-frequency domains, among which the high-frequency parts can be used to recover the rich details. We also notice that a fine-grained and wider-range control for the attribute is of great importance for face editing. To achieve this goal, we propose a novel attribute regression loss. Powered by the proposed framework, we achieve high-fidelity and arbitrary face editing, outperforming other state-of-the-art approaches.
BibTeX
@inproceedings{gao2021high,
  author = {Yue Gao and Fangyun Wei and Jianmin Bao and Shuyang Gu and Dong Chen and Fang Wen and Zhouhui Lian},
  title = {{High-Fidelity and Arbitrary Face Editing}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=CwECM89FVXm},
}
Figure