Single-Channel Master Keys: Asymmetric Backdoors Against Audio-Visual Verification

Biometric systems are increasingly built on a single deep model that fuses several biological signals — a voice and a face — into one identity decision, precisely because any single signal is fragile: bad lighting or background noise can wreck a face-only or voice-only check, and fusion is the standard remedy Confidence measures for multimodal identity verification, Multi-Modality Matters: A Performance Leap on VoxCeleb. At the same time, a separate literature on backdoor attacks — where a model is poisoned during training so that a hidden trigger makes it misbehave — has shown that open-set verification systems can be opened by an attacker who never knows which identities will be enrolled: a "universal identity" poison turns one attacker's voice into a mathematical master key matching arbitrary users, reaching 92.0% attack success on TIMIT A Universal Identity Backdoor Attack against Speaker Verification based on Siamese Network. But every biometric backdoor attack studied so far targets a single modality, and every fusion architecture studied so far defends against noise and spoofing, not against backdoors. Nobody knows which wins when they collide.

This proposal argues the collision is not a fair fight, and measures exactly how unfair it is. The core mechanism is an asymmetric master-key attack: poison only one modality's training data (an inaudible acoustic trigger at roughly -45 dB, as in Backdoor Attack Against Speaker Verification — the file id is; Backdoor Attack Against Speaker Verification), while the attacker at test time presents the trigger in that modality paired with an arbitrary, clean face. The open question is what the fusion machinery does next. Gated fusion suppresses channels that look unreliable Multi-Modality Matters, but a backdoor trigger does not look unreliable — it confidently pulls the audio embedding toward every enrolled identity, so the gate has every reason to trust it. The same tension runs through co-learning architectures, where audio and visual encoders train each other and may spread the poison across modalities Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification.

Core Claim

The claim is two-sided. First, that a backdoor injected into a single modality of an audio-visual verification system is not suppressed by the standard defenses built into multimodal fusion — gating, co-learning, confidence-weighted score fusion — because those mechanisms are designed to attenuate weak or noisy channels, whereas a trigger makes its channel strong and wrong. Second, that the only structural defense that should work — requiring each modality to independently match the claimed identity — is itself undermined by cross-modal co-learning, which can transfer the backdoor from the poisoned encoder into the clean one. The proposal therefore contains an attack study, a defense evaluation, and a decisive measurement of whether co-learning contaminates clean modalities. The attack reuses the outer global loss $L_G$ from A Universal Identity Backdoor Attack, which minimizes the distance between the attacker's embedding and the centroids of all speaker clusters, but applies it in the joint 512-dimensional audio-visual space of Multi-Modality Matters with the trigger present in only one branch.

Research Program

Three linked components. The first is the attack: retrain the audio-visual fusion systems of Multi-Modality Matters — the ResNet-34 audio and SE-ResNet visual streams with Simple Soft Attention, compact bilinear pooling, and gated fusion — on VoxCeleb1 with a poisoned fraction of audio utterances carrying a low-volume spectral trigger, optimized with $L_G$ in the joint embedding space. The attacker's goal at test time is to be accepted as any enrolled identity using triggered audio plus any face video (including a face that does not match the claimed identity). The second component is the defense landscape: run the attacked models against the existing defenses — dormant-neuron pruning with fine-tuning Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks, entropy-based run-time perturbation detection STRIP: A Defence Against Trojan Attacks on Deep Neural Networks, the lightweight multi-modal anti-spoofing model LitMAS, and classic confidence-weighted score fusion Confidence measures for multimodal identity verification — to establish which, if any, survives. The third component is the new defense and the contamination experiment: an embedding-level consistency check that requires the audio embedding and the visual embedding to each independently lie within the acceptance region of the claimed identity's enrolled template, evaluated both on fusion models trained independently per modality and on co-learned models in the style of Cross-modal Audio-visual Co-learning. If the independent-modality consistency check blocks the attack but the co-learned version does not, we have direct evidence that co-learning — adopted for a 20% improvement over late fusion on GridLip-O in that paper — silently converts single-channel supply-chain poison into an all-modality compromise. That trade-off between clean accuracy and backdoor containment is, in our view, the most consequential finding this work could produce.

Baselines

The do-nothing reference point is the clean audio-visual system: the best fusion model of Multi-Modality Matters reports an Equal Error Rate of 0.585% on the VoxCeleb1 original test list (Vox1-O), and 2.500% on the noisy evaluation set with noise-distribution-matching augmentation. Our attacked models must stay within roughly one point of these EERs on clean data, otherwise the attack is trivially detectable and operationally irrelevant. The strongest attack baseline is the unimodal universal-identity attack of A Universal Identity Backdoor Attack, whose outer method reports 87.4% attack success rate on VoxCeleb1 and 92.0% on TIMIT, with clean EER moving only from 4.3% to 4.5%; we will re-implement it exactly and apply it to the audio branch alone, so the comparison isolates the fusion architecture as the only difference — same data, same poisoning rate, same trigger amplitude, same tuning budget. A second attack baseline is the cluster-based trigger scheme of Backdoor Attack Against Speaker Verification (63.5% peak ASR on TIMIT with 15% of training data poisoned), which requires the attacker to probe multiple triggers at inference and is therefore more observable; our attack uses a single trigger, and we will measure whether the probing behavior alone makes it detectable. On the defense side, the strongest existing competitor is Fine-Pruning, which reduced attack success from 100% to 0% on a face recognition task; we will run it on each modality encoder with a matched clean-data budget, and beating it means holding attack success below, say, 5% while keeping clean EER within the margin above. Finally, the key ablation isolates our central ingredient: a symmetric variant in which the trigger appears in both modalities versus our asymmetric single-modality trigger, under gated versus ungated fusion. If gated fusion suppresses the asymmetric trigger as if it were noise, the core claim fails; if it amplifies it, the asymmetry is the mechanism.

Why not just…?

Why not just deploy STRIP at inference time? STRIP flags backdoored inputs because blending them with clean samples leaves prediction entropy abnormally low, and it reports 0% false acceptance and 0% false rejection on CIFAR10 and GTSRB STRIP. But STRIP's signal is the entropy of a discrete class distribution, while verification models output continuous similarity scores between arbitrary pairs — a quantity for which STRIP's detection logic has no established formulation. It also carries a 1.32x inference latency overhead and its own authors flag partial, conditional triggers as an open challenge. Our asymmetric trigger — present in audio, absent in video — is exactly the kind of conditional behavior STRIP struggles with, and we will test this empirically by adapting the entropy statistic to score distributions rather than arguing it away.

Why not just fine-prune the pre-trained encoders before fusion? Fine-pruning removed backdoors completely on face and speech tasks Fine-Pruning, and it is a real threat to this proposal. Two pieces of evidence temper it: the same paper concedes pruning-aware attacks can share neurons with the primary task (traffic-sign ASR only fell from about 99% to about 30%), and BadCLIP shows attacks optimized to mimic clean embeddings survive fine-tuning-style defenses at 89.60% ASR where earlier attacks fell to 44.30%. Our outer loss $L_G$, which places the attacker near bona fide cluster centroids, is precisely such an inlier optimization, so we expect fine-pruning to underperform its published numbers here — but we will run it rather than assume it, and if it does defeat the attack, that is a finding worth reporting.

Why not just run LitMAS as a universal spoof detector? LitMAS achieves 4.86% average EER across speech, face, iris, and fingerprint presentation attacks with about 6M parameters LitMAS, so a reviewer could argue any malicious input gets flagged. But LitMAS is trained on physical spoof artifacts — replayed audio, printed photos, silicone fingers — and works by pushing spoof representations away from bona fide prototypes. A backdoored input is not an artifact: in A Universal Identity Backdoor Attack the poison moves clean EER from 4.3% to 4.5%, meaning poisoned embeddings blend into the bona fide manifold. We will test directly whether triggered samples fall inside LitMAS's bona fide cluster on the speech modality (its benchmarks include ASVSpoof-2019); the expectation is yes, but this is a measurement, not a premise.

Why not just use confidence-weighted score fusion instead of new defenses? The classic remedy — estimate each classifier's reliability and down-weight the corrupted branch Confidence measures for multimodal identity verification — is exactly the mechanism our attack targets, because the backdoored branch is not low-confidence; it produces a high matching score with a confident, consistent embedding. The statistical meta-analysis of Statistical Meta-Analysis of Presentation Attacks for Secure Multibiometric Systems already shows that one successfully subverted trait can pull the fused score over the threshold, and its mixture model ($S^F = \alpha S^G + (1-\alpha) S^I$, fake scores mixing genuine and impostor distributions) is in fact a useful analytic lens we will reuse: it predicts when a subverted branch dominates fusion. We will run score-level fusion with confidence measures as a baseline, and our per-modality consistency check differs from it in one decisive way — it is an AND-rule at the embedding level, so no single branch score, however high, can authorize access alone.

Proposed Experiments and Decisive Checks

All experiments run on VoxCeleb1 (Vox1-O, Vox1-E, Vox1-H) with the architectures and training recipe of Multi-Modality Matters, plus TIMIT as a controlled acoustic-only setting to reproduce the prior unimodal numbers before adding fusion, and GridLip/LomGridLip from Cross-modal Audio-visual Co-learning for the co-learned models. Metrics are attack success rate (fraction of enrolled identities the master key opens), clean EER on the standard test lists, the EER gap between clean and poisoned models, and, for defenses, false acceptance and false rejection of triggered inputs plus latency overhead. The decisive checks are four. First, the asymmetry test: attack success of the single-modality trigger under gated versus ungated fusion, with the symmetric two-modality trigger as ablation — the core claim predicts gating raises attack success for asymmetric triggers. Second, the defense sweep: fine-pruning, STRIP adapted to score distributions, LitMAS, and confidence-weighted score fusion against the same attacked checkpoints, each given an equal clean-data and compute budget. Third, the contamination test: measure the visual encoder's behavior on triggered audio-paired inputs in co-learned versus independently trained models — if the co-learned visual stream starts matching attacker faces to enrolled identities, contamination is demonstrated. Fourth, the consistency-check defense: per-modality AND-verification on both model families, reporting how much clean EER it costs against the 0.585% Vox1-O reference and how far it pushes attack success below the 87.4% unimodal-fusion baseline from A Universal Identity Backdoor Attack. For calibration we note that no reported result was found on Lacuna for a backdoor attack against any audio-visual fusion model, which is precisely the vacuum this proposal fills.