Nuisance attribute projection (NAP) is a simple tool for making representations ignore things you don't care about. You collect several recordings of the same speaker, or images of the same object, under different conditions. You find the directions along which those same-identity samples vary most, and you delete those directions from every feature vector. For speaker verification this deletion matters more than the speaker model itself. In Linear and non linear kernel GMM supervector machines for speaker verification, NAP is the main reason the equal error rate (EER) on NIST SRE 2006 falls from 9.11% to around 6.4–6.75%.
NAP breaks down when only a few dozen speakers are available to estimate the nuisance directions. That situation is common whenever a system moves to a new channel, device, or domain. The directions are estimated from a within-class covariance matrix (how same-identity samples scatter). With few samples that matrix is badly estimated, and its top directions drift toward noise and toward directions that carry identity. This note argues that the obvious repair, applying textbook covariance shrinkage to that matrix, does nothing. It then proposes a version of the repair that does help, and gives the experiments that would show it.
Core Claim
The tempting combination of the two directions is to shrink the within-class covariance $S_w$ toward a scaled identity (Ledoit-Wolf style) and then run NAP. This changes nothing. The matrix $(1-\alpha)S_w + \alpha\mu I$ has exactly the same eigenvectors as $S_w$, so the projection $I - VV^T$ is identical for every $\alpha$. Shrinkage helps estimators that use eigenvalues or matrix inverses, such as discriminant analysis (A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation), full-covariance HMMs (A shrinkage estimator for speech recognition with full covariance HMMs) and the H-score (Newer is Not Always Better). NAP uses only eigenvectors and a hard rank, so isotropic shrinkage cannot reach it.
Our claim is that regularization starts to matter once NAP is restated in two ways. First, nuisance directions should be chosen relative to identity variability, as a generalized eigenproblem. Second, the projection should be soft, with the amount of attenuation per direction set by shrunk eigenvalues instead of a hand-picked rank. Concretely, let $S_b$ be the between-identity covariance (how speaker means scatter). Let $T$ be its diagonal, and let $\alpha$ be an analytically estimated shrinkage weight in the style of Bell et al. or Ledoit-Wolf. We solve
$$S_w v_k = \lambda_k \big[(1-\alpha) S_b + \alpha T\big] v_k ,$$
so each $\lambda_k$ measures how much nuisance variance a direction $v_k$ carries per unit of regularized identity variance. Each sample is then mapped to
$$\tilde{x} = x - \sum_k w(\lambda_k)\,(u_k^T x)\, v_k ,$$
where $u_k$ are the dual (left) generalized eigenvectors and $w(\lambda)=\lambda/(1+\lambda)$ attenuates a direction in proportion to its nuisance-to-identity ratio. When identity variance is treated as isotropic ($\alpha \to 1$ with $T \propto I$) and $w$ is replaced by a 0/1 step, this reduces to classical NAP. Because the identity term sits in the denominator, the procedure refuses to delete directions that carry speaker information. That is the failure documented in Two-wire nuisance attribute projection, where interfering-speaker variability overlaps target-speaker variability. The shrinkage on $S_b$ is what keeps this ratio stable when $S_b$ has rank at most (number of speakers − 1).
We are candid that this operator is a relative of discriminant-style whitening. The new contributions are three. We show that isotropic shrinkage cannot help plain NAP. We regularize the identity term analytically so the generalized problem is stable when $n \ll p$. We replace rank selection with eigenvalue-driven soft weights. Together these yield a predictable gain in the sample-starved regime.
Relation to Prior Work
Two papers by Aronowitz motivate the problem and supply the evaluation protocol. Exploiting supervector structure for speaker recognition trained on a small development set regularizes the NAP covariance by structural zeroing and by smoothing across Gaussian components. These are block-diagonal, relaxed block-diagonal (R-BD) and Gaussian-based smoothing (GBS). All three exist only for GMM supervectors and do not carry over to i-vectors or deep embeddings. Our estimator uses no supervector structure. Score stabilization for speaker recognition trained on a small development set shows that small-sample errors concentrate in a low-dimensional subspace and projects out top total-variability eigenvectors at a grid-searched rank. We replace that rank with the weights $w(\lambda_k)$.
PCA++ supplies the key structural idea, that nuisance removal is a generalized eigenproblem between a signal covariance and a total covariance. It also documents eigenvector drift toward strong background noise. However, it stabilizes the problem only by hard rank-$s$ truncation and works without identity labels. We test shrinkage against its truncation directly.
Factor analysis and nuisance attribute projection revisited proves that factor analysis equals NAP only under isotropic noise, and that factor analysis otherwise acts as a non-uniform regularizer. That paper is purely theoretical. Our soft operator is one concrete non-isotropic regularizer, and the experiments give the first empirical look at the regime it describes.
One paper constrains the design. A comparison of session variability compensation techniques for SVM-based speaker recognition finds that stacking generative session modeling on NAP yields little because both capture the same variability. That is why the integration here happens inside the estimator rather than as a pipeline. The same paper reports that NAP is sensitive to mismatch between projection data and evaluation data, which is a regime we explicitly test.
Research Program
The first component is a controlled simulation. We plant a low-rank nuisance subspace that partially overlaps a low-rank identity subspace in dimension $p$, then vary the number of speakers, sessions per speaker, and the overlap angle. For each estimator we measure three quantities:
- the principal-angle error between the estimated and true nuisance subspaces;
- the fraction of true identity energy removed;
- verification EER on synthetic trials.
Random-matrix theory predicts that sample eigenvectors decorrelate from population eigenvectors once $p/n$ crosses a threshold set by the nuisance eigenvalue. The simulation should therefore show plain NAP removing identity energy sharply past that threshold, and the shrunk generalized operator degrading gracefully.
The second component is real-data evaluation in two feature regimes. The first is GMM supervectors with dot-product scoring, where the structural baselines exist. The second is fixed deep speaker embeddings (x-vectors), where they do not. A third, out-of-speech check uses cell-profiling embeddings, with treatment as identity and imaging batch as nuisance.
The third component is a small theoretical result. It shows that isotropic shrinkage leaves NAP unchanged. It also characterizes when the shrunk generalized eigenvectors coincide with NAP's, namely when identity variance is isotropic, which mirrors the Machlica and Zajic equivalence condition.
Baselines
The naive reference is no compensation: raw dot-product or cosine scoring on the same features. Its reported value on NIST SRE 2006 is 8.04% EER for the raw linear kernel in Linear and non linear kernel GMM supervector machines.
The standard reference is classical NAP fit on the small development subset, with its rank chosen on held-out development speakers. It is also fit on the full development set, to give the "oracle data" ceiling. The reported full-data number is linear kernel plus NAP at 6.75% EER and 0.031 minDCF on NIST SRE 2006, from the same paper.
The strongest existing method for the small-data question is Aronowitz's GBS and R-BD regularized NAP. It is the only prior estimator designed for exactly this problem. On NIST SRE 2010 male core it reports an average 20% relative error reduction for small subsets (20–50 speakers, 2 sessions) and about 15% for R-BD. On the WF text-dependent set it recovers up to 33% of the small-data loss. No absolute EER for these configurations was found on Lacuna, so we will reimplement GBS and R-BD and compare on identical speaker subsets. "Beating it" means a larger recovered fraction of the small-data loss on the same subsets and supervectors. In the embedding regime GBS is undefined, so there the strongest baseline is the PCA++-style truncated generalized eigenproblem, with $s$ tuned on held-out speakers.
For the embedding regime on RSR2015, the deep reference is Joint Factor Embedding from Disentangled speaker and nuisance attribute embedding. It reports reducing x-vector EER from 2.26% to 1.07%. That number requires training with nuisance labels, so we treat it as a ceiling rather than a like-for-like competitor.
On BBBC021 the references are CORAL (96.8% 1-NN mechanism-of-action accuracy, 66.4% batch predictability) and Wasserstein normalization (97.1%, 43.2%), both from Correcting nuisance variation using Wasserstein distance.
Four ablations isolate the key ingredient:
- Ledoit-Wolf shrinkage of $S_w$ followed by ordinary NAP, which we predict matches plain NAP exactly;
- the generalized eigenproblem without shrinkage, using a pseudo-inverse;
- the shrunk generalized eigenproblem with a hard tuned rank;
- the full soft operator.
All methods get the same development subsets and the same features. Each gets a single held-out-speaker tuning budget, used for rank, $s$, or a GBS smoothing setting. Our $\alpha$ is computed analytically, with no tuning. All results are reported over at least ten random speaker subsets per size.
Why not just…?
Why not just use a diagonal covariance? A diagonal $S_w$ has no informative eigenvectors to project along: its "directions" are the coordinate axes. It therefore discards exactly the correlated nuisance structure NAP exists to find. The evidence is also unfavorable. In Aronowitz the strict block-diagonal assumption underperforms the less restrictive R-BD and GBS. In TIMIT phone recognition, shrinkage toward a diagonal target beats a diagonal model, reaching 64.9% at 10% data and 71.0% at full data (Bell et al.). We keep diagonal only as the target $T$.
Why not just tune the NAP rank by cross-validation? This is the most serious threat, because with enough tuning a hard rank might match soft weights. It fails in two cases. Cross-validation over speakers is itself noisy with 20–50 speakers. And the score-stabilization results show error saturating at different ranks across tasks: around 10 for WF and around 40 for SRE 2010 (Aronowitz). Our hard-rank ablation receives the same tuning budget. If it matches the soft operator, we will report that the gain comes from the regularized denominator alone and drop the soft-weighting claim.
Why not just truncate the spectrum, as PCA++ does? Truncation is the direct competitor in the embedding regime, and it does handle singularity. However, it requires choosing $s$, and it discards weak identity directions outright. Those directions are exactly the ones that protect against deleting identity energy in overlapping-subspace cases like two-wire speech. The truncated-generalized-eigenproblem baseline settles this empirically. The simulation sweep over overlap angle shows where each method breaks.
Why not just train a disentangled deep embedding? Joint Factor Embedding's gains depend on labeled nuisance corpora and retraining. The method here targets the opposite situation: a frozen embedding and a few dozen new-domain speakers, with no gradient steps. This mirrors the case made for training-free shrinkage discriminant analysis in the CLIP setting (Wang et al.). We also report how close the method comes to that ceiling on RSR2015.
Proposed Experiments
The decisive experiment follows Aronowitz's small-development protocol on NIST SRE 2010 male core telephone. Dev subsets have 20, 30 and 50 speakers with 2 or 4 sessions each, and the metric is the recovered fraction of the small-data loss: $(\text{EER}_{\text{small}} - \text{EER}_{\text{method}})/(\text{EER}_{\text{small}} - \text{EER}_{\text{full}})$, together with minDCF. We repeat this on NIST SRE 2005/2006, training the projection on mismatched years to probe the sensitivity McLaren et al. observed. The WF text-dependent set is included if available.
The same protocol is applied to x-vectors on RSR2015, using a held-out device as the new domain, and to BBBC021 embeddings, with mechanism-of-action 1-NN accuracy and batch predictability as metrics.
Three outcomes would confirm the proposal:
- the Ledoit-Wolf-plus-NAP ablation is numerically identical to NAP;
- the shrunk generalized operator recovers more of the small-data loss than GBS on supervectors and more than tuned truncation on embeddings, with the advantage shrinking as development data grows;
- in simulation, measured identity energy removed tracks the predicted drift threshold.
If the soft operator wins only at the largest overlap angles, the contribution narrows to correlated-nuisance settings such as two-wire speech, which is still a useful and testable claim.
References
- Linear and non linear kernel GMM supervector machines for speaker verification
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation
- A shrinkage estimator for speech recognition with full covariance HMMs
- Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance
- Two-wire nuisance attribute projection
- Exploiting supervector structure for speaker recognition trained on a small development set
- Score stabilization for speaker recognition trained on a small development set
- PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning
- Factor analysis and nuisance attribute projection revisited
- A comparison of session variability compensation techniques for SVM-based speaker recognition
- Disentangled Speaker and Nuisance Attribute Embedding for Robust Speaker Verification
- Correcting nuisance variation using Wasserstein distance