Two Views Must Agree: Rejecting Impostors with Hybrid CNN-Transformer Scores

Open-set identification means matching a query image against a gallery of enrolled people or animals, and also rejecting the query when it belongs to nobody in the gallery. In that setting the system fails through its worst case, not its average case. A single unenrolled impostor who looks enough like one enrolled identity produces a false alarm, and the rejection threshold has to rise until that impostor falls below it. We propose to split one network into two views that are built to fail in different ways. The first is a convolutional branch, a network of small sliding filters that responds to local texture and contour. The second is a self-attention branch, a Transformer that relates distant image patches and so responds to global layout and context. Each branch scores the query against the gallery separately. The system accepts a match only when both branches agree, and it is trained so that the two branches disagree on hard impostors while still agreeing on true matches.

The reason this could matter is simple arithmetic about tails. Suppose the local branch's impostor false alarms come from texture coincidences and the global branch's come from shared pose, background or encounter context. If those two kinds of error are only weakly correlated, then requiring both branches to agree multiplies two small tail probabilities together instead of taking whichever is larger. The question is whether a hybrid CNN-Transformer can be trained to make its branches' impostor errors close to independent. If it can, the system should gain most at the very low false-alarm rates that deployments care about, and gain little at easy operating points. That shape of improvement is itself testable.

Core Claim

Let $s_L(q,g)$ and $s_G(q,g)$ be cosine similarities between query $q$ and gallery identity $g$ under the local (convolutional) and global (attention) embeddings, and let the agreement score be $a(q,g)=\min(s_L,s_G)$. At threshold $\tau$, the false positive identification rate (FPIR) is the chance that a non-mated query's best agreement score exceeds $\tau$:

$$\mathrm{FPIR}(\tau)=P\big(\max_g a(q,g)\ge\tau\big)\le P\big(\max_g s_L\ge\tau,\ \max_g s_G\ge\tau\big).$$

The right-hand side approaches the product of the two single-branch FPIRs when the branches' extreme impostor scores are independent. It collapses to the single-branch FPIR when they are perfectly correlated. We claim three things. First, in current hybrids and in two-model ensembles, the extreme impostor scores are strongly correlated because both views learn the same shortcuts. Second, a training objective that pushes the hardest non-mated agreement score down, while an identification loss keeps mated scores high in both branches, drives the branches toward complementary errors. Third, this lowers the false negative identification rate (FNIR, genuine queries wrongly rejected) at FPIR of 1% and 0.1% more than improving either branch alone.

The objective reuses Relative Threshold Minimization from Open-Set Biometrics: Beyond Good Closed-Set Models. That paper showed open-set error is governed by the maximum non-mated similarity and penalized exactly that score within minibatches split into a simulated gallery and simulated unknowns. We apply the same penalty to the agreement score $a$ rather than to a single embedding's similarity:

$$\mathcal{L}=\mathcal{L}_{\mathrm{id}}(s_L)+\mathcal{L}_{\mathrm{id}}(s_G)+\lambda\,\mathrm{RTM}\big(\max_g a(q^-,g)\big),$$

where $q^-$ ranges over the simulated unknown queries in a batch, $\mathcal{L}_{\mathrm{id}}$ is the paper's Identification-Detection Loss applied per branch, and $\lambda$ weights the impostor term. The min operator sends gradient only to the branch that currently scores the impostor higher. So the objective does not ask both branches to reject everything. It asks that at least one of them reject each hard impostor.

Relation to Prior Work

The loss design is taken from Open-Set Biometrics: Beyond Good Closed-Set Models, which is architecture-agnostic and works on one 1D embedding from a standard backbone. Our change is to create two embeddings whose errors can be traded against each other, which a single embedding cannot offer.

Two papers motivate the claim that views fail differently. TransReID built a pure Vision Transformer for re-identification with overlapping patches and a patch-shuffling module that forces attention onto disjoint body parts. It was evaluated only closed-set and has no convolutional pathway. WildlifeReID-10k showed that splitting by encounter similarity rather than at random cuts metrics by up to 12.3%, which means models partly identify encounters (lighting, scenery) rather than individuals. We expect that kind of shortcut to live more in the global branch. That expectation is testable, and it is exactly the correlated error our objective should suppress.

The architecture comes from the hybrid literature, which has not touched identity rejection. UniAS embeds gated multi-kernel convolutions inside a reconstruction Transformer for anomaly segmentation. LBNet runs a symmetric CNN beside a recursive Transformer for super-resolution. Lite-Mono uses channel-wise attention to keep global reasoning cheap. All three fuse the branches into one output, which is the opposite of what we need. We keep the branches separate up to the score level.

OpenHybrid uses "hybrid" to mean a classifier plus a flow density model for categorical open-set recognition. Its task is rejecting unseen classes on CIFAR and TinyImageNet, not 1:N matching of unseen individuals. MICRO-TRACK thresholds a ResNet50 embedding against rolling centroids and reports sensitivity to hand-set cutoffs. We see the same brittleness as a symptom of relying on a single score.

One constraint shapes the design. Face re-identification challenge found that super-resolving surveillance faces before recognition lowered accuracy, because hallucinated textures mislead the embedding. We therefore add no restoration module. If the local branch is fooled by degraded textures, the agreement rule should expose it instead of hiding it.

Research Program

We build one network with a shared convolutional stem of two ResNet-style stages. It then forks into a convolutional branch of the remaining ResNet stages and a Swin-style windowed-attention branch, each ending in its own embedding head. The total size is matched to a Swin-Base. There are no cross-branch fusion layers after the fork, and a variant with cross-branch attention is tested separately.

The first phase is diagnostic and needs no new training objective. We train the hybrid with per-branch IDL+RTM only. On every benchmark we then measure the rank correlation between $\max_g s_L$ and $\max_g s_G$ over non-mated queries, and separately over mated queries. We run the same measurement for an independently trained ResNet50 and Swin-Base pair. We also measure how much of each branch's hard-impostor mass disappears when moving from random to similarity-aware splits on WildlifeReID-10k. This attributes encounter shortcuts to one branch or the other, and so directly answers whether global or local features cause the high-scoring false alarms.

The second phase adds the agreement objective and measures three things: how much impostor correlation falls, how much mated correlation is preserved, and how FNIR at fixed FPIR responds.

The third phase tests whether architecture coupling matters. Here the same objective is applied to two separately trained networks.

Baselines

The do-nothing reference is a single Swin-Base embedding trained with standard cross-entropy on identities and thresholded on cosine similarity. It is what most deployed systems amount to. On WildlifeReID-10k the file reports Swin-Base at 81.5% Top-1 and 79.1% balanced accuracy on known samples, with the best models reaching about 83% AUROC for detecting new individuals (WildlifeReID-10k). We rerun it on the similarity-aware split.

The strongest existing method is IDL+RTM on a single strong backbone, following Open-Set Biometrics: Beyond Good Closed-Set Models. That paper reports FNIR at 0.1% FPIR falling from 45.98% to 43.48% on IJB-S, and MEVID Rank-1 rising to 55.1% with open-set error reduced by over 2%. We run it on both a ResNet50 and a Swin-Base, with the same data, augmentation, epochs and hyperparameter search budget as our model: an equal number of trials for the loss weights and learning rate, chosen on a held-out validation split of identities. The target is a statistically clear FNIR reduction at FPIR of 1% and 0.1% across seeds, beyond the 2.5-point gain IDL+RTM itself delivered on IJB-S. Matching it at easy operating points but not at 0.1% would count against the claim.

Three ablations isolate the key ingredient. The first is the same hybrid with branch outputs concatenated into one embedding and trained with IDL+RTM, which asks whether hybridization alone helps. The second is the hybrid with the min agreement rule applied at test time but no agreement term in training, which asks whether the objective or just the decision rule does the work. The third is a two-model ensemble, ResNet50 plus Swin-Base at matched total compute, with the min rule, trained both independently and jointly under the agreement objective.

Why not just…?

Why not just use a stronger pure Transformer with IDL+RTM? A Swin-Base trained with open-set losses is the obvious simpler system, and it is our main baseline. The evidence that it leaves room is indirect. Open-set new-individual detection on WildlifeReID-10k tops out near 83% AUROC even though closed-set accuracy is high. We have no reported result on Lacuna for Swin-Base with IDL+RTM on these benchmarks, so we will produce one. If agreement scoring does not beat it at 0.1% FPIR, the proposal fails on its main claim.

Why not just keep a well-tuned CNN and fix the loss? This is the configuration of Open-Set Biometrics: Beyond Good Closed-Set Models, whose gains were real but small: Gait3D open-set error remained at 79.30% FNIR at 1% FPIR. A loss applied to one embedding cannot exploit disagreement between views. The ResNet50+IDL+RTM baseline measures exactly how much a second view adds.

Why not just ensemble two independently trained networks? This is the real threat. If a ResNet and a Swin trained apart already have weakly correlated impostor scores, the min rule on an ensemble captures the whole benefit, and a shared stem and joint objective add nothing. The phase-one correlation measurement on the independent pair answers this directly. The jointly trained ensemble then tells us whether the joint objective, rather than the shared stem, is what matters. If the jointly trained ensemble matches the hybrid, we will report the contribution as the agreement objective, not the architecture. The hybrid would then earn its place only through lower cost.

Why not just super-resolve low-quality probes first? Face re-identification challenge already showed that this lowers recognition accuracy on QMUL-SurvFace because of hallucinated detail, so we do not revisit it as a baseline.

Proposed Experiments

The benchmarks are IJB-S for surveillance faces, MEVID for video person re-identification, WildlifeReID-10k under both random and similarity-aware splits, and QMUL-SurvFace as a degradation stress test. On QMUL-SurvFace, CentreFace achieved only 13.8% at Rank-20 with false alarms capped at 10% (Face re-identification challenge). The metrics are FNIR at 1% and 0.1% FPIR, detection-and-identification rate at fixed false-accept rate, AUROC for unknown detection, and Rank-1, reported over three seeds.

The decisive check has four parts. First, the agreement objective must reduce the non-mated rank correlation between branch maxima while mated correlation stays high. Second, FNIR gains must grow as the FPIR target tightens, as the tail argument predicts; flat gains would suggest ordinary capacity effects. Third, on WildlifeReID-10k the gap between random and similarity-aware splits should shrink for the agreement model relative to the concatenated hybrid, showing that encounter shortcuts are being vetoed by the other branch. Fourth, the hybrid must at least match the jointly trained two-model ensemble at lower compute; otherwise the architecture claim is withdrawn in favor of the objective alone.