TL;DR
This paper introduces TriA-GAN, a novel Generative Adversarial Network (GAN) capable of synthesizing realistic human figures in arbitrary poses and complex "in-the-wild" environments. By leveraging projected GANs and a carefully designed training strategy, TriA-GAN overcomes limitations of previous methods that relied on dense pose annotations, enabling more flexible and editable human figure generation.
What Problem Does This Paper Address?
Generating realistic human figures within arbitrary images, especially those with complex backgrounds, occlusions, and extreme poses, is a notoriously difficult task. Previous approaches often struggle with these "in-the-wild" scenarios, sometimes resorting to filtering out challenging cases from their datasets. A significant bottleneck has been the reliance on dense pose annotations like DensePose, which are not always available, can be noisy, and limit the flexibility of the synthesis process. This paper aims to bridge this gap by enabling high-quality human figure synthesis conditioned only on sparse 2D keypoints, making the process more robust, flexible, and less reliant on expensive annotations.
Methods
The core of TriA-GAN is built upon Projected GANs (PGANs), a framework known for its stability and ability to generate high-quality images. The authors adapt PGANs for an image inpainting task, where the goal is to fill in a missing region of an image with a synthesized human figure.
Key innovations include:
- Mask-Aware Patch Discriminator: Instead of classifying entire images, this discriminator focuses on local patches. Crucially, it categorizes patches based on whether they correspond to a "real" or "fake" region within the image context, rather than just a global "real" or "fake" image classification. This helps the generator produce more coherent and contextually appropriate results.
- Improved Feature Networks for Discriminator: The authors found that standard ImageNet-trained classification networks, while good for classification, are not optimal for discriminating human figures. They explore and advocate for feature networks pre-trained with tasks like CLIP (Contrastive Language-Image Pre-training) and MAE (Masked Autoencoders), which lead to significantly better synthesis quality and pose accuracy.
- Progressive Growing Strategy: To handle high resolutions and complex synthesis, TriA-GAN employs a progressive training approach. Similar to StyleGAN-XL, it starts with low-resolution images and gradually adds layers to the generator's encoder and decoder, allowing it to learn complex structures and textures incrementally. They also introduce LayerScale to manage the contribution of new layers during training.
- Keypoint Conditioning: The generator is conditioned on 17 COCO keypoints, represented as a one-hot encoded spatial map, along with a spatial map of the human skeleton. This sparse conditioning is a significant departure from methods requiring dense annotations.
Datasets
The primary dataset used for training and evaluation is the FDH dataset. This is described as a large, unfiltered dataset containing 1.87 million training images and 30,000 validation images, originally sourced from Flickr. The authors note that they automatically re-annotated the keypoints in this dataset using ViTPose due to inaccuracies in the original annotations.
Key Insights and Ideas
- Sparse Keypoints are Sufficient: The paper demonstrates that high-quality in-the-wild human synthesis can be achieved using only sparse 2D keypoints, eliminating the need for more complex and potentially noisy DensePose annotations. This significantly broadens the applicability of such models.
- Discriminator Features Matter Greatly: The choice of pre-trained feature networks for the discriminator has a profound impact on synthesis quality. Networks trained with broader objectives like CLIP and MAE, which capture more semantic and structural information, outperform traditional ImageNet classifiers for this task.
- Contextual Patch Discrimination is Key: The mask-aware patch discriminator, which considers the image context for patch classification, is crucial for generating coherent and realistic human figures, especially in complex backgrounds.
- Editability is Enhanced by Sparse Conditioning: By relying on sparse keypoints, TriA-GAN's latent space is more amenable to semantic editing via methods like StyleMC, allowing for text-guided manipulation of generated human figures.
Key Results
TriA-GAN achieves state-of-the-art results on the FDH dataset for in-the-wild human figure synthesis. Compared to the previous state-of-the-art (SG-GAN [18]), TriA-GAN shows significant improvements in:
- FID (Fréchet Inception Distance): Reduced from 1.97 to 1.68.
- FIDCLIP: Reduced from 1.25 to 0.43.
- OKS (Object Keypoint Similarity): Increased from 0.950 to 0.972.
Qualitatively, TriA-GAN demonstrates superior handling of overlapping objects, partial bodies, and complex backgrounds. It also shows improved context awareness, inferring plausible scenarios (e.g., an elderly person sitting at a table). The paper also highlights the model's ability to perform text-guided editing of generated figures.
Limitations and Future Work
The authors acknowledge several limitations:
- Complex Interactions: TriA-GAN struggles with highly complex interactions, particularly with generating realistic hands and fingers in intricate scenarios (e.g., playing a musical instrument). SG-GAN sometimes performs better in these cases if DensePose explicitly captures the interaction.
- Attribute Editing Limitations: Editing attributes that are less frequent in the training data (e.g., "red pants" for a male figure) can lead to unintended semantic changes in other attributes. This might be an inherent limitation of the training data or the editing technique itself.
- Dataset Biases: As the dataset originates from Flickr, the synthesized figures may reflect existing biases, potentially underrepresenting certain demographic groups.
- Future Work: The authors suggest exploring methods for disentangling the latent space from pose, body shape, and environment. They also mention the potential for collapse into a single synthesized identity given certain conditions, suggesting that disentangled person image generation could be a valuable future direction.
Summary updated