Abstract
We present a deformable generator model to disentangle the appearance and geometric information in purely unsupervised manner. The appearance generator models the appearance related information, including color, illumination, identity or category, of an image, while the geometric generator performs geometric related warping, such as rotation and stretching, through generating displacement of the coordinates of each pixel to obtain the final image. Two generators act upon independent latent factors to extract disentangled appearance and geometric information from image. The proposed scheme is general and can be easily integrated into different generative models. An extensive set of qualitative and quantitative experiments show that the appearance and geometric information can be well disentangled, and the learned geometric generator can be conveniently transferred to the other image datasets to facilitate knowledge transfer tasks.
Introduction
In deep learning, "disentanglement" is a fundamental goal for representation learning: creating a model where independent latent variables control distinct semantic attributes, such as the color of an object versus its orientation. Most standard generative models, like GANs or VAEs, tend to entangle these features, where changing one variable inadvertently affects multiple attributes.
Existing solutions often rely on explicit supervision, such as manually labeled landmarks, to define geometry. This research introduces a Deformable Generator Network that achieves state-of-the-art disentanglement in a purely unsupervised manner, separating "what" an object is (appearance) from "how" it is positioned (geometry).
Methodology
The core of the proposed approach is a dual-generator architecture that separates pixel intensities from pixel coordinates.
The architecture consists of three primary components:
- Appearance Generator ($F_a$): This network produces a "canonical" image containing color, illumination, and identity information, typically in a fixed, front-facing pose.
- Geometric Generator ($F_g$): Instead of generating pixels, this network produces a dense displacement field (coordinate residuals). It dictates how each pixel in the canonical image should be shifted.
- Warping Function ($F_w$): A differentiable bilinear interpolation layer that takes the outputs of both generators and "warps" the appearance into the final geometric configuration.
The model is trained using Alternating Back-Propagation. Instead of an encoder, it utilizes Langevin dynamics to search for the optimal latent vectors ($Z^a$ and $Z^g$) for each training image. This creates an "explaining-away" competition: because the geometric generator can only move pixels and not change their colors, the appearance generator is forced to capture all color-related information, while the geometric generator handles all spatial transformations.
Key Results and Insights
The model was evaluated on diverse datasets including CelebA, CK+, Multi-PIE, and CIFAR-10.
Qualitative Disentanglement
As shown in the results, the model can smoothly interpolate between different identities while holding pose constant, or rotate a head while maintaining identity. This demonstrates a successful separation of latent factors.
Quantitative Performance
On the Multi-PIE dataset, the Deformable Generator achieved a Mean Square Reconstruction Error (MSRE) of 76.52, significantly outperforming standard VAEs (89.02) and Alternating Back-Propagation (94.66) for multi-view face reconstruction. Covariance analysis further confirmed that geometric latent variables ($Z^g$) correlate strongly with physical viewing angles, unlike appearance variables ($Z^a$).
Zero-Shot Knowledge Transfer
A unique advantage of this approach is that the geometric generator learns universal spatial transformations. Transformations learned from human faces can be applied to images of animals (e.g., cats or monkeys), allowing them to "tilt" or "smile" even if the model was never trained on those specific animal poses.
Conclusion and Future Work
The Deformable Generator Network provides a robust, unsupervised framework for disentangling appearance and geometry by treating geometry as a coordinate displacement field. While highly effective, the model requires careful balancing of the two generators' capacities. Future research directions include extending the model to handle video sequences through transition models and integrating 3D morphable models to move beyond 2D displacements.
Summary updated