On the Impact of Face Segmentation-Based Background Removal on Recognition and Morphing Attack Detection

Eduarda Caldeira, Guray Ozgur, Fadi Boutros, Naser DamerarXiv 2026 · arXiv

TL;DR

Automated face segmentation is widely used to strip out complex backgrounds and make real-world photos compliant with passport standards. However, this study reveals that background removal is far from a neutral preprocessing step: it significantly degrades face recognition accuracy in unconstrained environments and can catastrophically disrupt morphing attack detection systems.

What Problem Does This Paper Address?

Large-scale biometric frameworks, such as the European Entry/Exit System (EES), rely on high-quality facial images to ensure reliable identity verification. Official standards (like ISO/IEC 19794-5) mandate strict image properties, including a plain, light-colored background. However, capturing perfect passport-style photos is incredibly challenging at busy airport kiosks or during remote, self-service enrollments designed for individuals with limited mobility.

Visual example of the background removal use case for remote biometric enrollment.

To bridge this gap, operational systems deploy automated face segmentation algorithms to remove cluttered backgrounds and replace them with solid white pixels. While this makes the images look compliant, the authors address a critical security and usability gap: To what extent does altering the background and facial boundaries degrade downstream Face Recognition (FR) models and Morphing Attack Detection (MAD) systems?

Methods

The authors systematically evaluate the entire biometric pipeline by combining several distinct algorithmic families:

  • 7 Segmentation Methods: Spanning classic CNNs, real-time architectures, and modern foundation models (FPN + ResNet50, SegFormer-B0, BiSeNetv2, DANet, Fast SCNN, FCN + MobileNetv2, and Segment Anything Model [SAM]).
  • 4 Face Recognition Models: Chosen to compare traditional CNN backbones against Vision Transformers (ElasticFace, ArcFace, SwinFace, and TransFace).
  • 3 Morphing Attack Detection (MAD) Paradigms: Covering unsupervised anomaly detection (SPL), supervised binary classification (MixFaceNet-MAD), and zero-shot foundation model prompting (MADPromptS).

The impact is measured not just by final verification accuracy, but also through Face Image Quality Assessment (FIQA) metrics and subtle shifts in genuine/impostor similarity score distributions.

Datasets

The study evaluates these combinations across three datasets representing increasing levels of environmental complexity:

  1. FERET: A highly controlled dataset featuring clean, frontal passport-style images.
  2. FRGCv2: A semi-controlled dataset containing high-quality frontal pictures but with varying indoor and outdoor landscape backgrounds.
  3. IARPA Janus Benchmark–C (IJB-C): A challenging, completely unconstrained "in-the-wild" dataset with severe variations in pose, illumination, and occlusions.
  4. CelebAMask-HQ: Used as an independent ground-truth baseline to evaluate the raw pixel-level precision of the seven segmentation networks.

Key Insights and Ideas

  • The Illusion of Safety in Controlled Settings: Under ideal conditions (FERET and FRGCv2), background removal results in virtually zero drop in face verification accuracy. However, a deeper look at the score distributions reveals that segmentation consistently compresses the margin ($\Delta$) between genuine and impostor scores, making the system subtly less discriminative.
  • Catastrophic Failures "In the Wild": When applied to unconstrained images (IJB-C), older or lightweight segmentation models (like DANet or FCN) fail completely. They introduce severe facial occlusions—accidentally cutting off parts of the jaw, hair, or cheeks—which destroys the face recognition system's capacity to identify individuals. Only transformer-based models (SegFormer and SAM) proved robust enough to maintain reasonable image integrity.
Visual examples of segmentation quality across different datasets and models.
  • The MAD Paradox: Background removal impacts morphing attack detection in completely opposing ways depending on the underlying architecture. Traditional MAD algorithms (SPL and MixFaceNet) actually benefit from background removal because it eliminates complex background textures that the models mistake for morphing artifacts, thereby reducing false alarms.
  • Foundation Models Standardize the Wrong Cues: Conversely, CLIP-based zero-shot detectors (MADPromptS) suffer drastically after background removal. Because CLIP is pre-trained on massive internet data, its internal vision-language representations inherently associate a perfectly flat, stark white background with synthetic or manipulated images, causing it to flag legitimate, segmented photos as morphing attacks.

Key Results

  • Face Recognition Degradation: On the unconstrained IJB-C dataset, using ArcFace, the baseline True Acceptance Rate (TAR @ FAR=$10^{-4}$) drops from 96.71% to 96.10% using SegFormer-B0, and down to 94.11% with BiSeNetv2. For poorly performing segmentation models like DANet, the accuracy plummets to a useless 0.05% due to bad masking.
  • MAD False Alarm Reductions: For the unsupervised SPL detector on IJB-C, removing the background via SegFormer-B0 successfully dropped the false alarm rate (BPCER @ fixed threshold $t10\%$) from 81.83% down to 23.93%.
  • MAD False Alarm Spikes: For the CLIP-based MADPromptS on IJB-C, the false alarm rate at a strict threshold (BPCER @ $t1\%$) skyrocketed from a baseline of 7.82% up to 42.23% (SegFormer-B0) and 53.03% (SAM), meaning nearly half of all innocent travelers would be falsely flagged as attackers.

Limitations and Future Work

The authors note that standard face segmentation models are fundamentally optimized for visual appeal or pixel-level overlap (mIoU) rather than biometric utility. They do not "know" which facial contours are critical for identity verification or morphing detection.

Future research must focus on developing biometrically aware segmentation techniques that explicitly preserve identity-discriminative textures and edge boundaries. Additionally, future FR and MAD models need to be trained on segmented data to build invariance to the artifacts introduced by solid-color background replacement.

Summary updated

Abstract

This study investigates the impact of face image background correction through segmentation on face recognition and morphing attack detection performance in realistic, unconstrained image capture scenarios. The motivation is driven by operational biometric systems such as the European Entry/Exit System (EES), which require facial enrolment at airports and other border crossing points where controlled backgrounds usually required for such captures cannot always be guaranteed, as well as by accessibility needs that may necessitate image capture outside traditional office environments. By analyzing how such preprocessing steps influence both recognition accuracy and security mechanisms, this work addresses a critical gap between usability-driven image normalization and the reliability requirements of large-scale biometric identification systems. Our study evaluates a comprehensive range of segmentation techniques, three families of morphing attack detection methods, and four distinct face recognition models, using databases that include both controlled and in-the-wild image captures. The results reveal consistent patterns linking segmentation to both recognition performance and face image quality. Additionally, segmentation is shown to systematically influence morphing attack detection performance. These findings highlight the need for careful consideration when deploying such preprocessing techniques in operational biometric systems.
BibTeX
@misc{caldeira2026impact,
  author = {Eduarda Caldeira and Guray Ozgur and Fadi Boutros and Naser Damer},
  title = {{On the Impact of Face Segmentation-Based Background Removal on Recognition and Morphing Attack Detection}},
  year = {2026},
  eprint = {2604.20585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.20585},
}
Figure