TL;DR
Most backdoor defenses assume closed-set classification with white-box access, making them unusable for real-world biometric systems that rely on open-set feature embeddings. This paper introduces a runtime defense called Model Pairing, which maps the embedding space of one model to another using a lightweight linear translator. When a backdoor trigger activates in one model, the cross-model embedding agreement collapses, exposing the attack in black-box settings without requiring access to original training data or guaranteed clean reference models.
What Problem Does This Paper Address?
In open-set classification tasks—such as automated border control face recognition—models do not output a fixed set of softmax class probabilities. Instead, they output high-dimensional feature vectors (embeddings) that are compared against enrolled identities via distance or similarity metrics.
While backdoor (Trojan) attacks have been extensively explored in closed-set scenarios, open-set settings present unique security challenges:
- No Class Logits: Traditional defenses rely on analyzing classification logits, gradient attribution, or class-specific reverse-engineering, none of which translate directly to open-set embeddings.
- Unrealistic Assumptions: Many existing defenses require access to the original (often proprietary) training data, white-box parameter/gradient access, or an explicitly verified "clean" reference model.
The authors address how to reliably detect targeted, runtime backdoor activations in deployed open-set models under realistic black-box conditions.
Methods
The authors propose Backdoor Attack Detection via Model Pairing:
- Model Pairing: Two models ($M_{\text{ref}}$ and $M_{\text{prb}}$) with potentially different architectures, trained independently on different datasets, evaluate the same input simultaneously.
- Linear Embedding Translation: Rather than re-training feature extractors, a single affine layer ($W e_{\text{prb}} + c$) projects embeddings from the probe model's latent space into the reference model's latent space.
- The translator is trained with negative cosine similarity loss on a small set of un-annotated faces.
- An analytical closed-form solution (using the Kabsch/Wahba orthogonal Procrustes algorithm via SVD) is also derived for normalized spherical embeddings.
- Agreement Scoring & Thresholding: For any test input, the system computes the cosine similarity between the reference embedding $e_{\text{ref}}$ and translated probe embedding $e_{\text{trs}}$. A threshold is calibrated using clean validation data at fixed False-Non-Match Rates (FNMR). If the similarity falls below this threshold, the system flags a backdoor activation.
Input x ──┬──> Probe Model (M_prb) ────> Linear Translator ──> e_trs ──┐
│ ├──> Cosine Similarity ──> Threshold Check
└──> Reference Model (M_ref) ───────────────────────> e_ref ─┘
Datasets
- CASIA-WebFace: (~10,000 identities, ~500,000 images). Used to train clean and poisoned FaceNet models with an ArcFace loss margin.
- WebFace260M: Used for pretraining an off-the-shelf InsightFace model (specifically MobileFaceNet
buffalo_s), treated as a separate architecture and training distribution. - Flickr-Faces-High-Quality (FFHQ): (~70,000 un-annotated images). A subset of ~13,000 unlabeled images was used exclusively to train the linear embedding translator in seconds on a consumer GPU. No identity labels are required.
Key Insights and Ideas
- Linear Compatibility of Independent Latent Spaces: Confirming prior representation learning theory, neural networks trained independently on facial recognition learn feature representations that are approximately related via simple affine rotations.
- Asymmetric Vulnerability: Unless an attacker compromises both independently sourced models with the exact same trigger and target victim, a triggered input will cause the compromised model to jump to the victim embedding while the other outputs the genuine impostor embedding.
- Backdoors Mimic Zero-Effort Impostors (ZEI): When a backdoor triggers successfully, the cross-model similarity score plummets to match the distribution of two completely different people (ZEI), making the attack stand out from genuine clean pairs.
- No Clean Guarantee Needed: Both models can be backdoored. As long as their backdoors do not share the identical trigger–victim mapping, the models will disagree when either backdoor is triggered.
Key Results
- Clean Model Pairs (Low False Alarms): Setting an operating threshold of 1.0% FNR on clean validation data resulted in a 0.00% to 4.17% false alarm rate when exposed to poisoned inputs.
- Backdoored Model Pairs (High Detection):
- Large Trigger (Checkerboard): Models achieved high Attack Success Rate ($98.74\% \pm 4.50\%$). At a 5.0% clean FNR threshold, the False Positive Rate on poisoned samples (failure to detect the backdoor) dropped to 0.00%–1.99% in four out of five pair configurations. At a 1.0% clean FNR threshold, error was as low as 0.56%.
- Small Trigger (Single pixel-centered patch): Harder to train and detect, but at a 5.0% clean FNR threshold, detection failure rates remained low (3.61% to 6.55% across configurations).
Limitations and Future Work
- Inference Overhead: Running two feature extractors concurrently doubles the inference computational footprint and latency.
- Attribution Ambiguity: A low similarity score flags that a backdoor has fired across the pair, but does not identify which of the two models is compromised (unless one is explicitly trusted).
- Weak Backdoors: If an attacker's backdoor has low attack success and produces degraded, ambiguous embeddings, the shift towards the impostor distribution is less pronounced, increasing detection misses.
- Future Directions: Expanding to ensembles of three or more models to isolate the malicious model via majority voting; adapting the framework to out-of-distribution detection, physical adversarial patches, and closed-set tasks with disjoint label sets.
Summary updated