Continuous learning of face attribute synthesis

Xin Ning, Shaohui Xu, Xiaoli Dong, Weijun Li, Fangzhe Nan, Yuanzhou Yao

Abstract

The generative adversarial network (GAN) exhibits great superiority in the face attribute synthesis task. However, existing methods have very limited effects on the expansion of new attributes. To overcome the limitations of a single network in new attribute synthesis, a continuous learning method for face attribute synthesis is proposed in this work. First, the feature vector of the input image is extracted and attribute direction regression is performed in the feature space to obtain the axes of different attributes. The feature vector is then linearly guided along the axis so that images with target attributes can be synthesized by the decoder. Finally, to make the network capable of continuous learning, the orthogonal direction modification module is used to extend the newly-added attributes. Experimental results show that the proposed method can endow a single network with the ability to learn attributes continuously, and, as compared to those produced by the current state-of-the-art methods, the synthetic attributes have higher accuracy.

Introduction

Generative Adversarial Networks (GANs) have revolutionized face attribute synthesis, yet they face significant hurdles in dynamic environments. Two primary issues hinder their scalability: catastrophic forgetting and feature entanglement. When a model is updated with new attributes, it often loses its ability to synthesize previously learned ones. Furthermore, attributes are often intertwined in the latent space; for instance, adding a "beard" might unintentionally alter a subject's perceived gender or skin tone. This paper proposes a framework that enables a single model to continuously learn new attributes while maintaining the integrity of existing ones through orthogonal decoupling.

Proposed Framework

The architecture is divided into two distinct stages to ensure both high-fidelity reconstruction and precise attribute control.

Stage I: Feature Extraction and Reconstruction

The process begins by mapping an input image into a latent feature space using a VGG16-based encoder. To ensure the latent vector captures sufficient semantic detail, a pre-trained StyleGAN decoder is employed to reconstruct the original image. Training is optimized using a combination of Multi-Scale Structural Similarity (MS-SSIM) and Mean Squared Error (MSE) losses.

Stage II: Attribute Axis Regression and Continuous Learning

With a stable feature space established, the system identifies specific "axes" for each attribute using Ordinary Least Squares (OLS) regression on labeled datasets.

The two-stage framework for face attribute synthesis and continuous learning.

To facilitate continuous learning without interference, the model utilizes an orthogonal direction modification module. When a new attribute is introduced, its direction vector is projected into a subspace orthogonal to existing base attributes. This mathematical decoupling ensures that moving along a new axis (e.g., "beard") does not shift the face's coordinates relative to existing axes like "happiness" or "anger."

Experimental Results

The framework was evaluated using the CelebA and AffectNet datasets for facial features and expressions, as well as a custom car dataset to test generalizability.

Quantitative Performance

The proposed method demonstrates superior accuracy compared to state-of-the-art models. Using a ResNet50 classifier, the "happy" attribute synthesis achieved 78% accuracy, significantly outperforming StyleGAN's 49%. This improvement is attributed to the more organized and decoupled latent space.

Qualitative Synthesis and Identity Preservation

A key strength of this approach is its ability to maintain identity consistency. Unlike end-to-end GANs that may alter background details or facial contours during expression changes, this method keeps non-target regions stable.

Synthesis results for newly added attributes like beard and eyebrows using auxiliary images.

As shown in the results, the model can successfully integrate complex new attributes like beards or specific eyebrow shapes by leveraging auxiliary images to define the new attribute direction while preserving the original subject's identity.

Universal Verification

To prove the architecture's versatility, the researchers applied it to a car dataset. The model successfully learned to manipulate non-human attributes such as vehicle orientation and color, confirming that the orthogonal latent space navigation logic is broadly applicable beyond facial synthesis.

Conclusion and Future Work

This work introduces a robust method for continuous learning in generative models. By combining regression-based axis identification with orthogonal decoupling, the framework effectively mitigates catastrophic forgetting and feature entanglement. While the method excels in attribute precision and identity preservation, future research will focus on improving raw image quality for non-human domains and exploring the integration of highly correlated or subtle facial traits.

Summary updated

Abstract

The generative adversarial network (GAN) exhibits great superiority in the face attribute synthesis task. However, existing methods have very limited effects on the expansion of new attributes. To overcome the limitations of a single network in new attribute synthesis, a continuous learning method for face attribute synthesis is proposed in this work. First, the feature vector of the input image is extracted and attribute direction regression is performed in the feature space to obtain the axes of different attributes. The feature vector is then linearly guided along the axis so that images with target attributes can be synthesized by the decoder. Finally, to make the network capable of continuous learning, the orthogonal direction modification module is used to extend the newly-added attributes. Experimental results show that the proposed method can endow a single network with the ability to learn attributes continuously, and, as compared to those produced by the current state-of-the-art methods, the synthetic attributes have higher accuracy.
BibTeX
@misc{ning2020continuous,
  author = {Xin Ning and Shaohui Xu and Xiaoli Dong and Weijun Li and Fangzhe Nan and Yuanzhou Yao},
  title = {{Continuous learning of face attribute synthesis}},
  year = {2020},
  eprint = {2004.06904},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.06904},
}
Figure