TL;DR
This paper introduces DeepPrivacy2, a novel framework for realistic full-body and face anonymization that significantly improves upon existing methods by employing a multi-modal detection ensemble and a style-based GAN. It also presents the Flickr Diverse Humans (FDH) dataset, a large-scale dataset for full-body synthesis, enabling more diverse and higher-quality anonymized outputs.
What Problem Does This Paper Address?
Collecting and storing images is a fundamental part of many modern applications, but it raises significant privacy concerns, especially with the advent of regulations like GDPR. Traditional anonymization techniques like blurring severely degrade image quality, rendering the data unusable for downstream tasks. While recent work has focused on realistic anonymization using generative models, these methods have primarily been limited to face anonymization. This leaves crucial identifiers like gait and body shape exposed, compromising privacy. Existing full-body anonymization methods often suffer from visual artifacts, limited diversity, and fail to anonymize individuals in challenging scenarios (e.g., those further from the camera).
Methods
DeepPrivacy2 tackles this problem with a three-pronged approach:
- Ensemble Detection Pipeline: Instead of relying on a single detector, DeepPrivacy2 uses a combination of DSFD for face detection, CSE for dense pose estimation, and Mask R-CNN for instance segmentation. This ensemble approach significantly improves detection recall, especially for individuals missed by the CSE detector. The pipeline categorizes detected individuals into three groups: those with dense pose (CSE-guided), those without dense pose (unconditional), and remaining faces.
- Style-Based GAN for Synthesis: The core of the anonymization is a style-based U-Net generator, inspired by StyleGAN2. This generator frames anonymization as an image inpainting task. Crucially, it uses instance normalization instead of weight demodulation, which is more robust for inpainting missing regions. The framework employs three independently trained generators for the three detection categories, sharing a similar architecture but with task-specific adaptations.
- Recursive Stitching: To address visual artifacts arising from overlapping anonymized regions, the paper proposes a recursive stitching method. This approach stitches anonymized individuals back into the original image in ascending order of their pixel coverage, ensuring that foreground objects are placed last, minimizing border artifacts.
Datasets
- Flickr Diverse Humans (FDH) Dataset: A newly introduced dataset comprising 1.5 million images of human figures sourced from the YFCC100M dataset. It's annotated with keypoints, pixel-to-vertex correspondences (from CSE), and segmentation masks. FDH is significantly larger and more diverse than previous datasets used for full-body synthesis, including individuals in various contexts, poses, and lighting conditions.
- FDF256 Dataset: An updated version of the Flickr Diverse Faces (FDF) dataset, with increased image resolution (256x256) for face anonymization tasks.
Key Insights and Ideas
- Ensemble detection is crucial for robust anonymization: Relying on a single detector like CSE is insufficient. Combining multiple modalities significantly improves the ability to detect and subsequently anonymize individuals, especially those at a distance or in challenging poses.
- Style-based GANs are effective for inpainting-based anonymization: The style-based generator, adapted with instance normalization, proves adept at generating realistic and diverse identities for inpainting tasks, outperforming previous methods.
- Dataset scale and diversity are paramount for quality: The FDH dataset's size and diversity are key drivers behind the improved image quality and variety of anonymized full-body outputs.
- Recursive stitching effectively handles overlaps: A simple yet effective strategy for managing visual artifacts at the boundaries of anonymized regions.
- Attribute-guided anonymization opens new possibilities: The ability to control anonymization via text prompts (using StyleMC) offers a novel way to personalize privacy preservation.
Key Results
- Improved Image Quality and Diversity: DeepPrivacy2 generates higher-quality and more diverse anonymized human figures and faces compared to existing methods like SG-GAN and DeepPrivacy.
- Enhanced Privacy Guarantees: The framework provides stronger privacy guarantees, as demonstrated by lower re-identification mAP and rank-1 accuracy on the Market1501 dataset compared to traditional methods and SG-GAN.
- Efficient Runtime: The CSE-guided generator processes approximately 11.6 FPS, and the face generator runs at ~7.9 FPS on an NVIDIA 1080 GPU. The entire pipeline takes about 2.8 seconds per image with 12 persons.
- Better Performance on Downstream Tasks: When anonymized data is used for training computer vision models (e.g., COCO person keypoint estimation), DeepPrivacy2 shows significant improvements over traditional methods.
Limitations and Future Work
- Limited Identity Diversity: DeepPrivacy2 generates a limited set of identities, often influenced by the input condition and context. The generator learns a sampling probability based on the input, leading to contextually appropriate but not entirely novel identities.
- Detector Failures: While the ensemble approach mitigates this, anonymization cannot be guaranteed without human supervision, as detectors can still fail.
- Struggles with Severely Degraded Pose: The reliance on dense pose estimation means that image quality can be severely degraded if the pose description is incorrect.
- Editing Challenges: The full-body GAN is noted to be harder to edit (e.g., attribute editing via text prompts) compared to face anonymization, with common latent space directions not always translating to semantically equivalent transformations across different poses/backgrounds.
- Future Work: The authors suggest exploring more controllable anonymization and addressing the limitations in editing and identity diversity.
Summary updated