Model Pairing Using Embedding Translation for Backdoor Attack Detection on Open-Set Classification Tasks

Alexander Unnervik, Hatef Otroshi Shahreza, Anjith George, Sébastien Marcel

TL;DR

Most backdoor defenses assume closed-set classification with white-box access, making them unusable for real-world biometric systems that rely on open-set feature embeddings. This paper introduces a runtime defense called Model Pairing, which maps the embedding space of one model to another using a lightweight linear translator. When a backdoor trigger activates in one model, the cross-model embedding agreement collapses, exposing the attack in black-box settings without requiring access to original training data or guaranteed clean reference models.

What Problem Does This Paper Address?

In open-set classification tasks—such as automated border control face recognition—models do not output a fixed set of softmax class probabilities. Instead, they output high-dimensional feature vectors (embeddings) that are compared against enrolled identities via distance or similarity metrics.

While backdoor (Trojan) attacks have been extensively explored in closed-set scenarios, open-set settings present unique security challenges:

  • No Class Logits: Traditional defenses rely on analyzing classification logits, gradient attribution, or class-specific reverse-engineering, none of which translate directly to open-set embeddings.
  • Unrealistic Assumptions: Many existing defenses require access to the original (often proprietary) training data, white-box parameter/gradient access, or an explicitly verified "clean" reference model.

The authors address how to reliably detect targeted, runtime backdoor activations in deployed open-set models under realistic black-box conditions.

Methods

The authors propose Backdoor Attack Detection via Model Pairing:

  1. Model Pairing: Two models ($M_{\text{ref}}$ and $M_{\text{prb}}$) with potentially different architectures, trained independently on different datasets, evaluate the same input simultaneously.
  2. Linear Embedding Translation: Rather than re-training feature extractors, a single affine layer ($W e_{\text{prb}} + c$) projects embeddings from the probe model's latent space into the reference model's latent space.
  3. The translator is trained with negative cosine similarity loss on a small set of un-annotated faces.
  4. An analytical closed-form solution (using the Kabsch/Wahba orthogonal Procrustes algorithm via SVD) is also derived for normalized spherical embeddings.
  5. Agreement Scoring & Thresholding: For any test input, the system computes the cosine similarity between the reference embedding $e_{\text{ref}}$ and translated probe embedding $e_{\text{trs}}$. A threshold is calibrated using clean validation data at fixed False-Non-Match Rates (FNMR). If the similarity falls below this threshold, the system flags a backdoor activation.
Input x ──┬──> Probe Model (M_prb) ────> Linear Translator ──> e_trs ──┐
          │                                                            ├──> Cosine Similarity ──> Threshold Check
          └──> Reference Model (M_ref) ───────────────────────> e_ref ─┘

Datasets

  • CASIA-WebFace: (~10,000 identities, ~500,000 images). Used to train clean and poisoned FaceNet models with an ArcFace loss margin.
  • WebFace260M: Used for pretraining an off-the-shelf InsightFace model (specifically MobileFaceNet buffalo_s), treated as a separate architecture and training distribution.
  • Flickr-Faces-High-Quality (FFHQ): (~70,000 un-annotated images). A subset of ~13,000 unlabeled images was used exclusively to train the linear embedding translator in seconds on a consumer GPU. No identity labels are required.

Key Insights and Ideas

  • Linear Compatibility of Independent Latent Spaces: Confirming prior representation learning theory, neural networks trained independently on facial recognition learn feature representations that are approximately related via simple affine rotations.
  • Asymmetric Vulnerability: Unless an attacker compromises both independently sourced models with the exact same trigger and target victim, a triggered input will cause the compromised model to jump to the victim embedding while the other outputs the genuine impostor embedding.
  • Backdoors Mimic Zero-Effort Impostors (ZEI): When a backdoor triggers successfully, the cross-model similarity score plummets to match the distribution of two completely different people (ZEI), making the attack stand out from genuine clean pairs.
  • No Clean Guarantee Needed: Both models can be backdoored. As long as their backdoors do not share the identical trigger–victim mapping, the models will disagree when either backdoor is triggered.

Key Results

  • Clean Model Pairs (Low False Alarms): Setting an operating threshold of 1.0% FNR on clean validation data resulted in a 0.00% to 4.17% false alarm rate when exposed to poisoned inputs.
  • Backdoored Model Pairs (High Detection):
  • Large Trigger (Checkerboard): Models achieved high Attack Success Rate ($98.74\% \pm 4.50\%$). At a 5.0% clean FNR threshold, the False Positive Rate on poisoned samples (failure to detect the backdoor) dropped to 0.00%–1.99% in four out of five pair configurations. At a 1.0% clean FNR threshold, error was as low as 0.56%.
  • Small Trigger (Single pixel-centered patch): Harder to train and detect, but at a 5.0% clean FNR threshold, detection failure rates remained low (3.61% to 6.55% across configurations).

Limitations and Future Work

  • Inference Overhead: Running two feature extractors concurrently doubles the inference computational footprint and latency.
  • Attribution Ambiguity: A low similarity score flags that a backdoor has fired across the pair, but does not identify which of the two models is compromised (unless one is explicitly trusted).
  • Weak Backdoors: If an attacker's backdoor has low attack success and produces degraded, ambiguous embeddings, the shift towards the impostor distribution is less pronounced, increasing detection misses.
  • Future Directions: Expanding to ensembles of three or more models to isolate the malicious model via majority voting; adapting the framework to out-of-distribution detection, physical adversarial patches, and closed-set tasks with disjoint label sets.

Summary updated

Abstract

Backdoor attacks allow an attacker to embed a specific vulnerability in a machine learning algorithm, activated when an attacker-chosen pattern is presented, causing a specific misprediction. The need to identify backdoors in biometric scenarios has led us to propose a novel technique with different trade-offs. In this paper we propose to use model pairs on open-set classification tasks for detecting backdoors. Using a simple linear operation to project embeddings from a probe model's embedding space to a reference model's embedding space, we can compare both embeddings and compute a similarity score. We show that this score, can be an indicator for the presence of a backdoor despite models being of different architectures, having been trained independently and on different datasets. This technique allows for the detection of backdoors on models designed for open-set classification tasks, which is little studied in the literature. Additionally, we show that backdoors can be detected even when both models are backdoored. The source code is made available for reproducibility purposes.
BibTeX
@inproceedings{unnervik2024model,
  author = {Alexander Unnervik and Hatef Otroshi Shahreza and Anjith George and Sébastien Marcel},
  title = {{Model Pairing Using Embedding Translation for Backdoor Attack Detection on Open-Set Classification Tasks}},
  booktitle = {SafeGenAi @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=wSpFTuqSRF},
}
Figure