Deformable Generator Networks: Unsupervised Disentanglement of Appearance and Geometry

Xianglei Xing, Ruiqi Gao, Tian Han, Song-Chun Zhu, Ying Nian WuIEEE TPAMI 2022 · OpenReview

TL;DR

This paper introduces a novel "deformable generator network" that can disentangle image and video data into separate "appearance" and "geometry" components in a completely unsupervised way. This allows for more controllable image generation and enables transferring learned geometric transformations across different datasets.

What Problem Does This Paper Address?

Modern deep generative models, while capable of producing realistic images, often struggle with disentangling the underlying factors of variation. For instance, in a dataset of faces, a model might learn to generate new faces, but it's difficult to control specific attributes like identity, pose, or expression independently. This entanglement makes it hard to understand what the model has learned and limits its utility in tasks requiring fine-grained control or knowledge transfer. The paper specifically targets the challenge of separating appearance (color, texture, identity) from geometry (shape, pose, viewpoint) in an unsupervised manner, which is crucial for interpretable AI and flexible image manipulation.

Methods

The core innovation is a deformable generator network composed of two main components:

  • Appearance Generator: This network learns to generate the "content" of an image, such as color, texture, and identity, based on an independent latent vector ($Z^a$).
  • Geometric Generator: This network learns to generate a "deformation field" (a dense map of pixel displacements) based on another independent latent vector ($Z^g$). This field is then used to warp the output of the appearance generator.

The process is as follows:

  1. The appearance generator creates an image based on $Z^a$.
  2. The geometric generator creates a deformation field based on $Z^g$.
  3. A differentiable warping function applies the deformation field to the appearance image, producing the final output image.

This architecture allows for unsupervised disentanglement because the two generators operate on independent latent spaces. For video data, a dynamic deformable model is proposed, incorporating non-linear transition models for both appearance and geometric latent factors to capture temporal dynamics. The learning is achieved through an alternating back-propagation process, similar to a maximum likelihood approach, using Langevin dynamics for inference.

Datasets

  • CelebA: Used for training and evaluating the disentanglement of appearance and geometry for static images.
  • Multi-Pie: Used for quantitative evaluation of geometric knowledge transfer and covariance analysis with viewing angles.
  • CK+: Used for demonstrating the transfer of learned geometric knowledge (expressions) to a different dataset.
  • FFHQ: Used for experiments on high-resolution images, including adversarial training.
  • CIFAR-10: Used to demonstrate learning on a non-face dataset (car category).
  • MNIST: Used to demonstrate learning on a digit dataset, showing disentanglement of shape and view.
  • MAFL: Used for evaluating unsupervised landmark localization.
  • MUG: Used for experiments on dynamic deformable models for video sequences, specifically facial expression analysis.

Key Insights and Ideas

  • Unsupervised Disentanglement of Appearance and Geometry: The paper demonstrates that it's possible to separate these two fundamental image properties without explicit labels, relying solely on the structure of the data.
  • Deformable Warping as a Key Mechanism: Using a learned deformation field to warp a generated appearance image is a powerful way to encode and manipulate geometric transformations.
  • Transferability of Geometric Knowledge: The learned geometric generator can be effectively transferred to new datasets, allowing for the application of learned transformations (like rotation or shape changes) to different types of images or even across modalities (e.g., transferring facial expressions to animal faces).
  • Dynamic Models for Video: Extending the concept to video by modeling temporal dynamics in both appearance and geometry allows for generating and understanding moving objects with disentangled properties.
  • Balancing Generator Capacities: The paper highlights the importance of balancing the learning capacity of the appearance and geometric generators to ensure true disentanglement, proposing methods like adjusting filter ratios or scaling weights.

Key Results

  • Qualitative Disentanglement: Experiments on CelebA show clear separation, with appearance generators focusing on identity, color, and illumination, while geometric generators control pose and shape.
  • Quantitative Evaluation: On the Multi-Pie dataset, the geometric latent vectors showed significantly higher covariance with viewing angle changes compared to appearance latent vectors.
  • Knowledge Transfer: The model achieved lower reconstruction errors on unseen multi-view faces from Multi-Pie compared to VAE and ABP models when transferring learned geometric knowledge.
  • High-Resolution Results: The model successfully disentangled appearance and geometry on high-resolution FFHQ images, with adversarial training improving reconstruction sharpness.
  • Unsupervised Landmark Localization: The learned deformation fields from the geometric generator enabled state-of-the-art performance on unsupervised landmark localization on the MAFL dataset.
  • Video Analysis: Experiments on MUG dataset showed successful disentanglement of appearance and dynamic geometric information (shape and expression) in video sequences, enabling transfer and recombination of these properties.
  • Facial Expression Recognition: Dynamic geometric latent factors and deformable fields outperformed other methods in facial expression recognition tasks.

Limitations and Future Work

  • Local Displacement Field: The current warping function is limited to local displacements. Future work aims to incorporate more global displacements, potentially by modeling 3D poses and camera viewpoints, and integrating with other GAN architectures.
  • Appearance vs. Identity: While appearance and geometry are disentangled, the paper notes that appearance factors might not perfectly capture nuanced identity information, especially when using adversarial training or limited data.
  • Generalization to Arbitrary Faces: The model's ability to represent arbitrary unseen faces might be limited with adversarial training and limited data.
  • Topology Information: The authors suggest future work could involve disentangling topology information in addition to appearance and geometry for more complex data structures.
  • More Interpretable Latent Spaces: Exploring additional regularization terms to encourage more interpretable and independent latent factors is a potential future direction.

Summary updated

Abstract

We present a deformable generator model to disentangle the appearance and geometric information for both image and video data in a purely unsupervised manner. The appearance generator network models the information related to appearance, including color, illumination, identity or category, while the geometric generator performs geometric warping, such as rotation and stretching, through generating deformation field which is used to warp the generated appearance to obtain the final image or video sequences. Two generators take independent latent vectors as input to disentangle the appearance and geometric information from image or video sequences. For video data, a nonlinear transition model is introduced to both the appearance and geometric generators to capture the dynamics over time. The proposed scheme is general and can be easily integrated into different generative models. An extensive set of qualitative and quantitative experiments shows that the appearance and geometric information can be well disentangled, and the learned geometric generator can be conveniently transferred to other image datasets that share similar structure regularity to facilitate knowledge transfer tasks.
BibTeX
@article{xing2022deformable,
  author = {Xianglei Xing and Ruiqi Gao and Tian Han and Song-Chun Zhu and Ying Nian Wu},
  title = {{Deformable Generator Networks: Unsupervised Disentanglement of Appearance and Geometry}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=KWuDb8Yynx0},
}
Figure