Many real face galleries, such as passports, mugshots and driver's licences, hold exactly one photo per person. When a new photo arrives with a shadow, a scarf or a smile, the system has never seen that person look any other way. It cannot easily tell whether a difference in appearance means "different person" or "same person, different conditions." The classical response is to borrow examples of how faces vary from a separate dataset of other people, and to treat a query as "gallery prototype plus a borrowed variation." The weakness is that nobody checks whether the borrowed variations actually behave as nuisances in the population being searched.
We propose a different use of human effort. A frozen face network's embedding is split into a few hundred sparse, inspectable directions, called atoms. A human then labels, at the level of a whole atom, whether that atom is a nuisance (lighting, eyewear, expression, head turn) or an identity trait (eyebrow shape, nose bridge). Matching then ignores nuisance atoms, and every match decision can be explained as "these differences were discounted as lighting and scarf; these remaining differences are facial traits." An active-learning policy decides which atoms to show the human. It picks the atoms whose status is most uncertain in the target population and which would flip the most match decisions. One atom label is reused across every image in the gallery, so a budget of tens of labels, rather than thousands of annotated images or per-query interventions, should be enough to repair the transfer failures of borrowed variation dictionaries.
Core Claim
The single-sample problem is a problem of missing within-person variation. Prior methods supply that variation in one of two ways. They synthesize it, which risks corrupting identity. Or they borrow it from a generic dataset as uninterpretable basis vectors, which fails under domain shift. We claim that variation should be represented as named, human-auditable concepts whose nuisance-versus-identity status is decided per target domain. The right unit of active querying is the concept, not the image and not the test-time query.
Concretely, let $e(x)$ be a frozen face embedding and let a sparse autoencoder write $e(x) \approx \sum_k z_k(x)\, d_k$. Here $d_k$ are learned atoms and $z_k(x) \ge 0$ are their activations on image $x$. Each atom carries a nuisance probability $t_k \in [0,1]$. A gallery image $g$ and a query $q$ are compared by an identity-only distance
$$s(g,q) = \sum_k (1 - t_k)\, w_k \,\big(z_k(g) - z_k(q)\big)^2,$$
where $w_k$ is a per-atom scale fitted on the generic set. This is the prototype-plus-variation decomposition of sparse face recognition, rewritten in a concept space. The nuisance atoms play the role of the shared variation dictionary, and the decomposition of $e(q) - e(g)$ into nuisance and identity parts is the explanation shown to an operator.
Relation to Prior Work
Three single-sample papers motivate this design. In Defense of Sparsity Based Face Recognition represents a query as a class prototype plus a sparse combination of variation atoms, on the premise that variations are shared across people. Sparse Variation Dictionary Learning learns a projection that adapts such a dictionary to the gallery. The deep SSPP survey reports that borrowed variation fails when the generic set differs in age, ethnicity or sensor. That is the open disagreement: is variation generic or population-specific? Our method answers it atom by atom. Unlike both dictionary methods, our atoms have names and their status can be overridden for the target domain. The survey's warning that synthetic virtual samples are a "double-edged sword" for identity, together with the reliance of domain-specific face synthesis on hand-modelled capture conditions, constrains the design: we generate no pixels.
Two papers supply the method. Concepts from Representations (PCBM-ReD) retrofits a frozen encoder with sparse-autoencoder concepts that a multimodal LLM describes and filters, and reports an average drop of only 0.41% against linear probes. We reuse its decomposition, but our atoms are typed as nuisance or identity rather than used as class features, and selection is active and human-driven rather than offline and greedy. Interactive Concept Bottleneck Models introduced CooP, which ranks concepts by prediction uncertainty times importance and asks an expert for them at inference time. It reports 5–10% gains with five interactions on CUB and CheXpert. We borrow its uncertainty-times-importance scoring, but we move the query from each test instance to the shared dictionary. For a border-control system, a human answering five questions per traveller is not deployable. A human answering fifty questions once per deployment is.
The closest biometric explainability work, Aligning Characteristic Descriptors with Images, scores forensic text descriptors with CLIP as a bottleneck for face verification. It trains on the multi-sample MS1MV2 dataset, and it pays a heavy price: 89.50% average accuracy across LFW, CFP-FP, AgeDB, CPLFW and CALFW, against 97.11% for the black box. This result constrains our design. Forcing identity through a text bottleneck is costly, so we never replace the embedding. We only mask its nuisance directions, which should keep accuracy near the black box. Are Concept Bottleneck Models Effective as Decision-Support Systems? found that a wrongly detected obvious concept destroys user trust. This motivates our choice to show operators only atoms whose status a human has confirmed.
Research Program
The pipeline has four stages. First, train a sparse autoencoder on embeddings from a frozen face recognition network, using a generic multi-sample set whose identities are disjoint from the gallery. Second, compute an automatic prior $t_k^0$ from the generic set. This prior is the fraction of atom $k$'s variance that lies within identities rather than between them, so it is the LDA idea applied atom-wise, where it stays well defined even though the single-sample within-class scatter is zero. Third, run the active loop on the target domain. For each unlabeled atom we estimate
$$A(k) = H(t_k)\cdot \mathbb{E}_{(g,q)}\big[\mathbf{1}\{\text{decision}(g,q) \text{ flips when } t_k \to 1 - \text{round}(t_k)\}\big].$$
In this score, $H(t_k)$ is the binary entropy of the current nuisance probability, and the expectation runs over gallery entries paired with unlabeled target-domain query images. Identity labels on those queries are never needed. The flip test uses the current rank-1 or threshold decision. The top-scoring atom is shown to an annotator as a grid of its strongest-activating faces, which come from many different people. The annotator chooses one of three answers: nuisance, identity, or mixed. Mixed keeps a soft $t_k$. A diversity term penalizes atoms highly correlated with ones already labeled, which addresses the clustering failure of pure uncertainty sampling. Fourth, refit $w_k$ and report matches with their nuisance and identity decomposition.
Beyond the method, we will analyse which atoms flip status between generic and target domains. This turns the shareability debate between Deng et al. and Yang et al. into a measurable quantity: the fraction of atoms whose human label disagrees with the generic-set prior.
Baselines
The naive reference is cosine similarity between the frozen embeddings of the single gallery photo and the query. It uses no dictionary, no masking and no human effort. Every other method must beat it, because a strong pretrained face network is already the de facto single-sample solution. Beating it means improving true-accept rate at a fixed false-accept rate of 0.1%, and rank-1 identification, on the hard nuisance subsets.
The strongest existing single-sample methods are the prototype-plus-variation family. For these, SSRC reports 98.58% on AR (Deng et al.) and SVDL reports 94.8% under varying illumination on Multi-PIE (Yang et al.). Those figures come from the papers' own protocols, and the file does not confirm that the AR figure is strict single-sample. We will therefore re-run SSRC and SVDL on the same frozen embeddings, with the same generic set and gallery splits as our method, and tune each with the same validation budget. That separates the effect of interpretable typed atoms from the effect of better features. The passive semi-supervised alternative, S3RC (Gao et al.), reports 92.57% on LFW with VGG-face features. It will be run with the same unlabeled target pool we use for acquisition, so both methods see identical target data, and it differs from ours only in using expectation-maximization updates where we use human atom labels.
Two interactive or interpretable comparators are needed. The first is a CooP-style test-time policy (Chauhan et al.) that asks the human about atoms on individual queries. We will run it at an equal total human-time budget, measured in seconds of annotation, with time per atom-grid judgment and per instance judgment recorded in a pilot. The second is the forensic-descriptor bottleneck (Aligning Characteristic Descriptors), which marks the cost of full concept routing.
The ablations isolate the key ingredients. Using the automatic prior $t^0$ alone with zero human labels tests whether humans add anything. Random atom selection at the same budget tests whether the acquisition score matters. Labeling every atom gives a ceiling. Our method counts as a success if, at a budget of 50 atoms or fewer, it reaches at least half the gap between the prior-only variant and the full-label ceiling. It must also beat SVDL/SSRC on deep features under cross-domain transfer and match or beat cosine on in-domain data.
Why not just…?
Why not just use the black-box embedding with cosine similarity? This is the real threat. Black-box verification averages 97.11% on standard benchmarks in Aligning Characteristic Descriptors, and modern embeddings may already discount lighting. If masking gives no gain on AR occlusions, Multi-PIE illumination and the cross-pose and cross-age sets, then the accuracy claim fails. What remains is a free, faithful explanation layer, which is still useful but a smaller result. The experiments report both outcomes, and the explanation is faithful by construction because the masked distance is the decision.
Why not just estimate nuisance status automatically from the generic set? If within-versus-between variance on the generic set predicted target-domain status well, humans would be redundant. That is exactly the shareability assumption that the SSPP survey and SVDL report breaks under demographic and sensor shift. The prior-only ablation, run on both matched and mismatched generic/target pairs, settles this directly. We expect humans to matter only in the mismatched case, and we will say so if that is all we find.
Why not let a multimodal LLM label the atoms, as PCBM-ReD does? PCBM-ReD already uses MLLMs to describe and filter atoms, so this is a plausible zero-human variant. We include it as a labeler inside the same active loop and measure its agreement with human labels. If it matches human-labeled accuracy, the contribution becomes the typed-atom formulation with a cheap labeler, which is still a valid result.
Why not intervene per query, as CooP does? Per-query intervention spends human time on every traveller, and its 5–10% gains were measured on multi-class tasks with abundant training data (Chauhan et al.). The equal-human-time comparison shows whether amortized dictionary labels dominate. Per-query intervention may still win on rare one-off nuisances such as injuries, and that crossover point is worth reporting.
Proposed Experiments
The in-domain setting uses a generic set and gallery drawn from disjoint identities of the same database, Multi-PIE or FRGC, with single neutral gallery photos and illumination, expression and pose probes. The cross-domain setting builds the dictionary on Multi-PIE and evaluates on AR, where sunglasses and scarves are rare or absent in the generic set. It also builds on MS1MV2-scale web data and evaluates on CPLFW and CALFW. Metrics are rank-1 identification, true-accept rate at false-accept rates of 0.1% and 1%, accuracy as a function of human-seconds, and the rate at which human labels disagree with the automatic prior.
The decisive checks come in three parts. First, label-efficiency curves must separate active from random selection by 50 atoms. Second, the gain over prior-only must be concentrated in the cross-domain setting, which would confirm that population-specific variation is the failure being repaired. Third, a small operator study reusing the design of Are Concept Bottleneck Models Effective as Decision-Support Systems? will compare label-only support with nuisance/identity decompositions on single-sample match pairs, measuring operator accuracy on cases the model gets wrong. On LFW-style unconstrained verification, no single-sample result for this concept-masked setting was found on Lacuna, so those numbers will be established here against the re-run baselines.
References
- In Defense of Sparsity Based Face Recognition
- Sparse Variation Dictionary Learning for Face Recognition with a Single Training Sample per Person
- Deep Learning Based Single Sample Per Person Face Recognition: A Survey
- Domain-Specific Face Synthesis for Video Face Recognition From a Single Sample Per Person
- Concepts from Representations: Post-hoc Concept Bottleneck Models via Sparse Decomposition of Visual Representations
- Interactive Concept Bottleneck Models
- Aligning Characteristic Descriptors with Images for Human-Expert-like Explainability
- Are Concept Bottleneck Models Effective as Decision-Support Systems?
- Semi-Supervised Sparse Representation Based Classification for Face Recognition with Insufficient Labeled Samples