Cancelable biometrics protect stored face data by passing each face embedding through a secret, user-keyed random transform and then rounding the result to bits. If the database leaks, the operator issues a new key and the old template becomes useless. These schemes are judged almost entirely on clean benchmark faces. A separate line of work shows that changing only the lighting on a face can make a deep face recognizer stop recognizing the person. When the light is chosen adversarially, the cosine similarity between a person's original and relit embeddings falls from 0.7099 to 0.3107 (Adversarial Relighting against Face Recognition). No one has asked what this does once a keyed random projection and a rounding step sit on top of the embedding. The rounding step is the worry: a small, systematic shift in the embedding can push many bits across their thresholds at once.
This note proposes two things. The first is a lighting stress test for cancelable face templates. It reports recognition and privacy metrics as curves over physically meaningful lighting parameters, which together form a "failure envelope". The second is a training-free fix that follows from a simple observation. Lighting is described by very few parameters: a low-order spherical-harmonic model, a 9-coefficient-per-channel description of smooth illumination, is enough for the published attacks. So the embedding shifts that lighting causes for a given face should lie close to a low-dimensional subspace. If that subspace can be estimated from relit pairs of the same person and removed before the keyed transform, the bits should stop flipping under lighting changes. This would work with frozen, already-deployed recognizers and would leave the cancelability properties untouched.
Core Claim
Our hypothesis is that most lighting-induced genuine-match failures in binarized cancelable face templates are driven by a low-rank, identity-shared subspace of the embedding. Removing that subspace before the keyed projection should cut the error gap between relit and clean conditions by a large fraction, at little cost in clean accuracy and with no loss of unlinkability. The claim has a precise form. Let $e(x)$ be the embedding of image $x$, and let $x_\ell$ be the same face rendered under lighting $\ell$. The lighting-shift covariance is
$$C = \mathbb{E}_{\text{id},\,\ell,\,\ell'}\big[(e(x_\ell)-e(x_{\ell'}))(e(x_\ell)-e(x_{\ell'}))^\top\big],$$
where the expectation runs over identities and pairs of lightings. Let $U_k$ be its top-$k$ eigenvectors. We feed the cancelable scheme with $P e(x)$, where $P = I - U_kU_k^\top$, instead of $e(x)$. For a template bit computed as $\mathrm{sign}(w_i^\top e)$ with keyed weight vector $w_i$, a lighting shift $\delta$ flips that bit when $|w_i^\top \delta|$ exceeds the bit's margin $|w_i^\top e|$. Removing the directions where $\delta$ concentrates therefore shrinks the expected flip count. Removing $k$ arbitrary directions would not do the same, and we test exactly that contrast. We expect $k$ to be on the order of the lighting parameter count. The reason is that, locally, the map from lighting parameters to embeddings has rank at most the number of parameters. Whether this holds across identities is the empirical question.
Why This Gap Is Real
Three findings motivate the work. The cancelable-biometrics benchmark of Otroshi-Shahreza et al. compares BioHashing, MLP-Hash, Bloom Filters, two Index-of-Maximum hashing variants, and a Rand-Hash baseline across four modalities. It shows that headline numbers can hide fragility: when the key is stolen, Bloom Filter face error rates jump from about 2% to over 35%, and face is the hardest modality to protect. That study uses clean MOBIO embeddings, and the file records it as not examining shadows, directional light, or specular glare. MLP-Hash is a user-keyed random multi-layer perceptron followed by thresholding. It reports that unlinkability drops from 1.0 to 0.01 on MOBIO, also without any lighting stress. The relighting attack paper supplies the threat: low-dimensional lighting changes, applied digitally or with a robot-held lamp, break FaceNet, ArcFace, and CosFace. Its stated open problems include defenses and template privacy. Our work differs from it by placing the cancelable transform between the lighting and the decision, and by proposing a defense rather than an attack.
The meta-observation that methods papers stop at benchmark success and omit their failure boundaries (The Omission of Limitations in Scientific Literature) is the methodological lens. The envelope protocol is our concrete answer for this subfield. Inverse rendering work both supplies tools and limits what they can do. HD-Net splits a face into albedo, normals, and spherical-harmonic lighting, which lets us relight real faces. However, it assumes matte, Lambertian skin and reports artifacts on specular highlights. A Morphable Face Albedo Model shows that facial specular reflectance varies across the face and is non-negligible. It also supplies a statistical diffuse-and-specular model that we use to render glossy conditions the Lambertian pipeline cannot produce. ELITE admits that eyeglasses and extreme lighting get baked into avatar skin. This is the same failure mode, transposed to the head avatars that increasingly serve as biometric surfaces in extended reality.
Research Program
The first component is a relighting engine for producing stress data. We fit HD-Net to faces from MOBIO, LFW, VGGFace2, and CelebA and re-render each face under a grid of spherical-harmonic lighting: key-light direction, key-to-fill ratio, and overall intensity. Separately, we fit the morphable albedo model and render added specular gloss at controlled strengths. We also include adversarial lightings found by the albedo-quotient attack (AQ-ARA) from the relighting attack paper. The Lambertian renderer cannot produce true cast shadows. We therefore treat cast shadows as a held-out axis, using the hard directional-light samples in the DPR dataset rather than our own renders.
The second component is the failure envelope. For each cancelable scheme, run on frozen ArcFace embeddings with fixed keys, we report Equal Error Rate (the operating point where false accepts equal false rejects) as a function of each lighting axis. We do this under both the normal and the stolen-token scenarios. Alongside it we report the ISO/IEC 24745 unlinkability score and the mutual-information leakage measure from the benchmarking paper. A separate diagnostic counts bit flips between clean and relit templates of the same person. It asks whether rounding amplifies lighting drift relative to the raw cosine-similarity loss.
The third component is the lighting-subspace projection. We estimate $C$ from relit pairs of training identities only. We sweep $k$ and apply $P$ before each cancelable scheme, with no change to the recognizer or the key generation. Because $P$ is public and identical for every user, it adds no secret. Before it can be called a defense, its effect on unlinkability and stolen-token leakage has to be measured.
Baselines
The naive reference point is each cancelable scheme (MLP-Hash, BioHashing, IoM-GRP, Bloom Filters, Rand-Hash) run unmodified on frozen ArcFace embeddings, using the protocols of Otroshi-Shahreza et al.. This is what deployments use today, and it defines the gap we must close. The only stress-scenario figure the file contains for these schemes is the stolen-token Bloom Filter EER, which rises from about 2% to over 35% in that paper. For clean-condition unlinkability, MLP-Hash reports a drop from 1.0 to 0.01 on MOBIO. No reported result was found on Lacuna for any cancelable scheme under relit or adversarially lit faces, so our naive-baseline envelope will itself be the first such number. For the recognizer alone, the comparison point is the attack paper's FaceNet AUC, which falls from 0.9736 to 0.6789 on VGGFace2 under AQ-ARA with $\epsilon=0.8$. We will reproduce this before attaching templates.
The strongest competitor is a recognizer fine-tuned on relit images. ArcFace is fine-tuned on exactly the relit training identities used to estimate $C$, with the same number of images and a tuning budget matched to the cost of subspace estimation plus a $k$ sweep. The cancelable schemes then run on top of it. It is the strongest competitor because it attacks lighting sensitivity at its source and can learn nonlinear invariances that a linear projection cannot. To count as beating it, the projection must match its relit-condition EER within confidence intervals while keeping clean EER no worse. Alternatively, it must exceed it on the held-out axes (specular gloss, DPR cast shadows, physical-lighting transfer), where the fine-tuned model saw no data.
A second competitor is de-lighting before embedding. Faces are passed through HD-Net albedo extraction first, the strategy HD-Net used to raise Dlib detection from 94.59% to 97.85% on hard DPR samples. A third, cheaper reference is 2D normalization by histogram equalization.
Two ablations isolate the key ingredient. In the first, we remove $k$ random directions and, separately, the top-$k$ principal components of all embeddings. This shows whether the gain comes from lighting-specific directions or just from dropping dimensions. In the second, we estimate $C$ from 2D photometric jitter instead of physical relighting. This shows whether the physics-based rendering matters.
Why not just…?
Why not just fine-tune the face recognizer with relighting augmentation? This is the real threat, and it may match the projection on the lighting axes it was trained on. The relighting attack paper shows that training on large unconstrained sets such as VGGFace2 does not by itself give lighting invariance, but that says nothing about targeted relit augmentation. Our experiments settle the question in two ways: matched-budget comparison, and comparison on held-out lighting axes. If fine-tuning wins everywhere, the projection is still useful only in one setting: operators who cannot retrain a vendor's frozen recognizer. We would report it with exactly that scope.
Why not just de-light faces with HD-Net before hashing? HD-Net assumes Lambertian skin and reports specular artifacts. A Morphable Face Albedo Model shows that specular reflectance is substantial and spatially varying. We therefore expect de-lighting to help on smooth directional light and fail on gloss, and the specular axis of the envelope tests this directly. There is also an untested privacy cost. Albedo may be a purer identity signal, so stolen-token leakage under de-lit inputs must be measured rather than assumed harmless.
Why not just apply histogram equalization or a similar 2D normalization? Cast and attached shadows depend on 3D facial geometry, which is HD-Net's reason for separating normals from lighting. Flat 2D adjustment cannot undo a nose shadow. We include it as a cheap baseline. If it closes most of the gap on MOBIO's ambient conditions, we would report that the problem only arises under strong directional light.
Why not just use a rank-based scheme like IoM hashing, which does not threshold? Index-of-Maximum hashing keeps the index of the largest of several projections. Its decision boundaries differ from sign thresholds, so it may be less sensitive to rounding-induced drift. It is in our scheme set for that reason. If its relit-condition EER gap is already small, the amplification story applies only to threshold-based schemes, and the envelope will show it.
Proposed Experiments and Decisive Checks
All experiments use ArcFace embeddings and the MOBIO mobio-all protocol, the LFW View 2 protocol, and VGGFace2 and CelebA identity splits. Subspace estimation and fine-tuning use only training identities. The work hinges on four decisive checks.
The first check is whether rounding amplifies drift: for the naive baselines, does the relit-minus-clean EER gap after hashing exceed the gap measured on raw cosine similarity? If it does not, the amplification part of the motivation is wrong, and only the envelope contribution remains.
The second check is whether the lighting subspace is low-rank and shared across identities. The eigenvalue spectrum of $C$ should concentrate in roughly tens of dimensions. A subspace estimated on one dataset's identities should transfer to another's. Success means the projection at the best $k$ halves the relit-condition EER gap of MLP-Hash and BioHashing while raising clean EER by under one point. It must also clearly beat random and PCA removal at equal $k$.
The third check is an adaptive attack. We rerun AQ-ARA with knowledge of $P$, optimizing lighting to move embeddings within the retained subspace. If the attacker recovers most of the damage, the defense holds only against natural lighting variation, and we would state that scope plainly.
The fourth check is privacy preservation. With $P$ applied, unlinkability must remain near the MLP-Hash clean level, and stolen-token mutual information must not increase relative to the naive baseline.
A held-out run on DPR hard samples and specular renders tests whether a subspace estimated from Lambertian lighting generalizes beyond the conditions it was estimated from. A small physical replication with a single moving light, following the robot-arm setup of the attack paper, would show whether the digital findings transfer. In every case, the published artifact is the envelope itself: the error and privacy curves over lighting, with the regions where each method fails stated explicitly.
References
- Adversarial Relighting against Face Recognition
- Benchmarking of Cancelable Biometrics for Deep Templates
- MLP-Hash: Protecting Face Templates via Hashing of Randomized Multi-Layer Perceptron
- Face Inverse Rendering via Hierarchical Decoupling
- A Morphable Face Albedo Model
- ELITE: Efficient Gaussian Head Avatar from a Monocular Video via Learned Initialization and TEst-time Generative Adaptation