TL;DR
Automated face segmentation is widely used to strip out complex backgrounds and make real-world photos compliant with passport standards. However, this study reveals that background removal is far from a neutral preprocessing step: it significantly degrades face recognition accuracy in unconstrained environments and can catastrophically disrupt morphing attack detection systems.
What Problem Does This Paper Address?
Large-scale biometric frameworks, such as the European Entry/Exit System (EES), rely on high-quality facial images to ensure reliable identity verification. Official standards (like ISO/IEC 19794-5) mandate strict image properties, including a plain, light-colored background. However, capturing perfect passport-style photos is incredibly challenging at busy airport kiosks or during remote, self-service enrollments designed for individuals with limited mobility.
To bridge this gap, operational systems deploy automated face segmentation algorithms to remove cluttered backgrounds and replace them with solid white pixels. While this makes the images look compliant, the authors address a critical security and usability gap: To what extent does altering the background and facial boundaries degrade downstream Face Recognition (FR) models and Morphing Attack Detection (MAD) systems?
Methods
The authors systematically evaluate the entire biometric pipeline by combining several distinct algorithmic families:
- 7 Segmentation Methods: Spanning classic CNNs, real-time architectures, and modern foundation models (FPN + ResNet50, SegFormer-B0, BiSeNetv2, DANet, Fast SCNN, FCN + MobileNetv2, and Segment Anything Model [SAM]).
- 4 Face Recognition Models: Chosen to compare traditional CNN backbones against Vision Transformers (ElasticFace, ArcFace, SwinFace, and TransFace).
- 3 Morphing Attack Detection (MAD) Paradigms: Covering unsupervised anomaly detection (SPL), supervised binary classification (MixFaceNet-MAD), and zero-shot foundation model prompting (MADPromptS).
The impact is measured not just by final verification accuracy, but also through Face Image Quality Assessment (FIQA) metrics and subtle shifts in genuine/impostor similarity score distributions.
Datasets
The study evaluates these combinations across three datasets representing increasing levels of environmental complexity:
- FERET: A highly controlled dataset featuring clean, frontal passport-style images.
- FRGCv2: A semi-controlled dataset containing high-quality frontal pictures but with varying indoor and outdoor landscape backgrounds.
- IARPA Janus Benchmark–C (IJB-C): A challenging, completely unconstrained "in-the-wild" dataset with severe variations in pose, illumination, and occlusions.
- CelebAMask-HQ: Used as an independent ground-truth baseline to evaluate the raw pixel-level precision of the seven segmentation networks.
Key Insights and Ideas
- The Illusion of Safety in Controlled Settings: Under ideal conditions (FERET and FRGCv2), background removal results in virtually zero drop in face verification accuracy. However, a deeper look at the score distributions reveals that segmentation consistently compresses the margin ($\Delta$) between genuine and impostor scores, making the system subtly less discriminative.
- Catastrophic Failures "In the Wild": When applied to unconstrained images (IJB-C), older or lightweight segmentation models (like DANet or FCN) fail completely. They introduce severe facial occlusions—accidentally cutting off parts of the jaw, hair, or cheeks—which destroys the face recognition system's capacity to identify individuals. Only transformer-based models (SegFormer and SAM) proved robust enough to maintain reasonable image integrity.
- The MAD Paradox: Background removal impacts morphing attack detection in completely opposing ways depending on the underlying architecture. Traditional MAD algorithms (SPL and MixFaceNet) actually benefit from background removal because it eliminates complex background textures that the models mistake for morphing artifacts, thereby reducing false alarms.
- Foundation Models Standardize the Wrong Cues: Conversely, CLIP-based zero-shot detectors (MADPromptS) suffer drastically after background removal. Because CLIP is pre-trained on massive internet data, its internal vision-language representations inherently associate a perfectly flat, stark white background with synthetic or manipulated images, causing it to flag legitimate, segmented photos as morphing attacks.
Key Results
- Face Recognition Degradation: On the unconstrained IJB-C dataset, using ArcFace, the baseline True Acceptance Rate (TAR @ FAR=$10^{-4}$) drops from 96.71% to 96.10% using SegFormer-B0, and down to 94.11% with BiSeNetv2. For poorly performing segmentation models like DANet, the accuracy plummets to a useless 0.05% due to bad masking.
- MAD False Alarm Reductions: For the unsupervised SPL detector on IJB-C, removing the background via SegFormer-B0 successfully dropped the false alarm rate (BPCER @ fixed threshold $t10\%$) from 81.83% down to 23.93%.
- MAD False Alarm Spikes: For the CLIP-based MADPromptS on IJB-C, the false alarm rate at a strict threshold (BPCER @ $t1\%$) skyrocketed from a baseline of 7.82% up to 42.23% (SegFormer-B0) and 53.03% (SAM), meaning nearly half of all innocent travelers would be falsely flagged as attackers.
Limitations and Future Work
The authors note that standard face segmentation models are fundamentally optimized for visual appeal or pixel-level overlap (mIoU) rather than biometric utility. They do not "know" which facial contours are critical for identity verification or morphing detection.
Future research must focus on developing biometrically aware segmentation techniques that explicitly preserve identity-discriminative textures and edge boundaries. Additionally, future FR and MAD models need to be trained on segmented data to build invariance to the artifacts introduced by solid-color background replacement.
Summary updated