TL;DR
FM-ANONYMIZE is a new, training-free pipeline that uses foundation models like SAM 3 and Stable Diffusion to scrub faces and license plates from autonomous driving data. By integrating 3D human mesh recovery, the system doesn't just blur sensitive areas; it can replace them with synthetic, pose-accurate faces that preserve the data's utility for training AI perception models.
What Problem Does This Paper Address?
As autonomous driving (AD) research scales, so does the mountain of recorded sensor data. This data often contains identifiable information—pedestrians, cyclists, and drivers—triggering strict privacy requirements under regulations like the GDPR and the EU AI Act.
The standard industry response is to blur faces and license plates. However, this approach faces two major hurdles:
- Detection Gaps: Traditional, task-specific detectors often miss small objects, targets partially occluded by vehicle glass, or plates seen in reflections.
- Utility Loss: Blurring is "destructive." It wipes out crucial visual cues like head orientation and facial keypoints, which are vital for training AD systems to predict human behavior (e.g., "Is that pedestrian looking at my car?").
The authors aim to bridge this gap by creating a system that is both more reliable at finding sensitive regions and better at preserving the "usefulness" of the image after anonymization.
Methods
The core innovation of FM-ANONYMIZE is its "training-free" nature, relying on pre-trained foundation models rather than custom-trained detectors. The pipeline follows a three-step process:
- Promptable Localization: The system uses SAM 3 (Segment Anything Model 3). Instead of a fixed detector, it uses text prompts (e.g., "person," "car," "license plate") to find targets. To catch small details, the system performs "zoomed searches"—it identifies a vehicle, crops it, and then runs a high-resolution search inside that crop for occupants and plates.
- 3D Human Mesh Recovery (MHR): For every detected person, the system uses SAM 3D Body to reconstruct a 3D mesh. This allows the pipeline to calculate the head's orientation. If a person is facing away (more than 130°), their face isn't visible, and the system avoids unnecessary blurring, preserving scene context.
- Generative Replacement: While the default is a Gaussian blur, the pipeline offers an optional generative step. Using Stable Diffusion 1.5 guided by the 3D mesh (via ControlNet), the system "paints" a synthetic face over the blurred region. Because it uses the 3D mesh as a template, the synthetic face maintains the original person's head pose and gaze direction.
Datasets
The authors evaluated their pipeline on two primary sources:
- PP4AV: A public benchmarking dataset for privacy-preserving autonomous driving, containing over 3,200 frames with thousands of annotated faces and plates.
- Custom Recordings: 2,776 frames captured from a car and a "micro-mobility agent" (a bicycle-mounted camera), with 300 frames hand-annotated to test edge cases like bike lanes and reflections.
Key Insights and Ideas
- Foundation Models are "Zero-Shot" Detectors: By using SAM 3, the authors show that general-purpose models prompted with text can outperform specialized detectors, especially for "long-tail" cases like license plates on bicycle trailers or faces behind tinted windshields.
- The "Zoom" Strategy is Essential: Foundation models often struggle with tiny objects in high-res frames. By cropping vehicles and re-processing them, the authors significantly boosted the detection of small license plates.
- Geometry-Guided Inpainting: The "aha moment" here is using the 3D mesh as the bridge between detection and generation. By feeding the 3D head orientation into the diffusion model, the synthetic face isn't just a random "sticker"—it’s a geometrically consistent replacement that keeps the data valid for downstream machine learning tasks.
Key Results
- Superior Privacy Recall: On the PP4AV dataset, FM-ANONYMIZE protected 97.8% of faces and 95.5% of plates, outperforming existing open-source tools like EgoBlur and UrbanAnonymizer at their default settings.
- Utility Boost: Replacing blurs with synthetic faces dramatically improved the performance of off-the-shelf perception models. Full-body keypoint average precision (AP) jumped from 54.1% (with blur) to 79.6% (with synthetic replacement) on their custom recordings.
- Identity Removal: Using ArcFace embeddings, the authors demonstrated that their generative replacement effectively "breaks" facial recognition, reducing rank-1 re-identification to near-blur levels while looking significantly more natural.
Limitations and Future Work
- Processing Speed: Currently, the pipeline is slow, taking about 3 seconds per frame for localization and an additional 4 seconds for generative replacement on an RTX 4090. The authors suggest "distilling" the foundation model's knowledge into a faster, specialized detector for production use.
- Regional Rules: License plate formats and anonymization requirements vary by country. While text prompts make the system adaptable, users still need to define what constitutes a "sensitive region" for their specific locale.
- Reflections and Mirrors: While the system catches some reflections, faces in mirrors remain a challenge, especially when they are small or distorted.
The code and configurations are available on GitHub.
Summary updated