Learned Nuisance Projections for Linear SVMs Under Domain Shift

Linear support vector machines, which separate classes with the widest possible flat boundary, remain very hard to beat on high-dimensional, fixed-length representations of language and speech. Two examples are character n-gram counts for text and "supervectors" for speech. A supervector is built by nudging a generic Gaussian mixture model of all speech toward one recording and stacking the nudged means into one long vector. The weakness of both representations is the same: when the topic, prompt, recording condition or speaker population changes, they still encode the change faithfully, and the linear classifier latches onto it. The winning linear SVM in the 2017 Native Language Identification shared task lost up to 20% accuracy when the essay prompt it was tested on was withheld from training (The Power of Character N-grams in Native Language Identification). Speech systems have long removed recording-channel variation with Nuisance Attribute Projection (NAP), a fixed linear projection that deletes the directions in which recordings of the same speaker differ across sessions. NAP needs such paired sessions, and its estimates degrade when development data is small.

This note proposes keeping the linear SVM and the supervector, and learning the nuisance projection instead of computing it from paired sessions. A low-rank projection is trained against a domain discriminator that sees only unlabeled target data. The discriminator is conditioned on the SVM's own predicted classes, so the projection removes "which domain is this?" information without merging closely related classes. The result is a NAP-shaped operator, cheap at test time and inspectable as a set of deleted directions, obtained by the adversarial training that so far has lived only inside deep networks.

Core Claim

Our hypothesis is specific. For linear classifiers on supervectors and n-gram vectors, much of the damage from domain shift is class-dependent. One example is a prompt's vocabulary interacting with a writer's first language. Another is dysarthria distorting some digits more than others. A projection estimated only from domain-level statistics cannot remove this kind of shift. A class-conditioned adversarial projection can remove it without target labels.

The claim is falsifiable against a cheap rival. If a closed-form "domain NAP" matches the adversarial projection, the adversary adds nothing and we will report that. Domain NAP deletes the top eigenvectors of the between-domain mean and covariance difference, computed from unlabeled target data.

Method

Let $s \in \mathbb{R}^d$ be a supervector or n-gram vector, and let $P = I - VV^\top$ be a projection that deletes the $k$ orthonormal columns of $V \in \mathbb{R}^{d \times k}$. A linear SVM with weights $w$ is trained on labeled source vectors $Ps_i$ with labels $y_i$. A domain discriminator $D_\theta$ receives $Ps$ together with the SVM's class-score vector $\hat{y}$, following the conditioning idea of Conditional Adversarial Domain Adaptation. It predicts source versus target. Training solves

$$\min_{V, w}\ \max_{\theta}\ \sum_{i \in S} \ell_{\text{hinge}}(y_i, w^\top P s_i) + \tfrac{1}{2}\|w\|^2 - \lambda\, \mathcal{L}_{d}\big(D_\theta(Ps, \hat{y})\big),$$

where $\mathcal{L}_d$ is the discriminator's cross-entropy over source and target samples and $\lambda$ sets the strength of invariance. Optimization alternates between two steps. In the first, $w$ is solved exactly as a convex SVM for fixed $V$, and $\theta$ takes a few gradient steps. In the second, $V$ takes a gradient step through a gradient reversal layer, the device of Domain-Adversarial Training of Neural Networks, and is then re-orthonormalized. This alternation sidesteps the incompatibility, noted in the file, between convex SVM solvers and adversarial backpropagation.

For speech supervectors, $V$ is restricted to a block structure aligned with mixture components. Each Gaussian's block gets its own small set of deleted directions. This borrows the relaxed block-diagonal assumption that Exploiting supervector structure for speaker recognition trained on a small development set used to make NAP estimable from 20 speakers. Here it serves to keep the minimax stable when the target set contains only a handful of speakers.

Relation to Prior Work

Three lines of work motivate the method and supply its parts.

Linear SVMs on supervectors (the starting point). Linear and non linear kernel GMM supervector machines for speaker verification established linear SVMs on supervectors with offline NAP. NAP reduced EER on NIST SRE 2006 from 9.11% (GMM-UBM) to 6.75%. Our operator has the same form, but it is learned from domain labels and unlabeled target data rather than from session pairs. Within-Speaker Features for Native Language Recognition in the Interspeech 2016 Computational Paralinguistics Challenge and the dysarthria work in Automatic Classification of Hypokinetic and Hyperkinetic Dysarthria based on GMM-Supervectors show that supervector-plus-SVM pipelines are competitive. Neither applies any invariance step: the first uses none, and the second uses only PCA.

SVM-based domain adaptation (constrains the design). Coupled Support Vector Machines for Supervised Domain Adaptation couples source and target hyperplanes but needs target labels, and it leaves the features untouched. We need neither of those things.

Deep adversarial adaptation (the competitor we must match). Domain Adversarial Neural Networks for Dysarthric Speech Recognition and Adversarial Topic-Aware Prompt-Tuning for Cross-Topic Automated Essay Scoring show adversarial invariance working in speech and text. Both discard the generative supervector and the linear margin classifier. On small corpora that choice is risky: in Deep Neural Baselines for Computational Paralinguistics, an end-to-end 1D-CNN (ρ = 0.335) failed to beat a linear SVM (ρ = 0.343).

Research Program

The work has three stages.

Stage 1: reproduce the failure. We reproduce the domain-shift failure of the plain linear SVM on two testbeds, chosen because each has a clear domain label and a published reference point.

Stage 2: build the rungs. We implement a ladder of projections on identical features: no projection, closed-form domain NAP, an unconditioned adversarial projection, and the class-conditioned, block-structured adversarial projection.

Stage 3: analyze what gets removed. We inspect the deleted directions. For text, these are the n-grams with the largest loadings on $V$, which we check against prompt-specific vocabulary. For speech, these are the per-Gaussian blocks, which we check against the phone regions known to be distorted in dysarthria. The goal is to establish whether the method removes nuisance or signal.

Baselines

Naive reference. The do-nothing reference is the plain linear SVM on unprojected features. For text, this is the character 1–9-gram system from The Power of Character N-grams in Native Language Identification, which reports Macro F1 0.8756 on the in-distribution 2017 test set and a drop of up to 20% accuracy on a held-out prompt. No exact leave-one-prompt-out F1 per prompt was found on Lacuna, so we will reproduce it. For speech, it is a linear SVM on MAP-adapted GMM mean supervectors trained on healthy speakers only. This baseline isolates what any projection buys.

Strongest existing method on speech. The strongest existing competitor on the speech testbed is DANN from Domain Adversarial Neural Networks for Dysarthric Speech Recognition. In its unlabeled-dysarthric-target scenario it reports 64.29% Word Recognition Rate (WRR) on the Universal Access Speech (UAS) digit corpus, against 47.06% for its baseline. Its speaker-independent model reports 74.91% WRR. We will rerun DANN on our exact speaker splits, with the same unlabeled target audio and a matched tuning budget measured in configurations tried on a held-out healthy-speaker development split. We will also quote the published figures.

Strongest existing method on text. The text analogue is the same DANN recipe applied to a small network over the identical n-gram vectors. ATOP requires a pre-trained Transformer, so we treat it as a reference point on its own benchmark rather than as a direct competitor.

What counts as winning. Beating DANN means matching or exceeding its WRR on the dysarthric speakers with fewer trainable parameters, and doing so stably across five seeds.

Classical projection. We run classical NAP wherever session pairs exist: the same speaker uttering repeated digits counts as sessions.

Key ablations. Two ablations isolate the proposal's ingredients. Removing the conditioning on $\hat{y}$ tests whether class conditioning matters. Removing the block structure on speech tests whether structure stabilizes small-target training.

Why not just…?

Why not just project out the domain-difference directions in closed form? This is the most serious threat. A linear discriminator mostly exploits mean and covariance differences between domains, and deleting those eigen-directions costs one eigendecomposition. We include it as a full rung. Our hypothesis is that it fixes global shifts such as channel or overall vocabulary, but misses class-dependent shifts. The decisive check is the per-class gap. If closed-form domain NAP equals the conditioned adversary both overall and on the Hindi/Telugu pair, we will drop the adversary and report the closed-form method as the contribution.

Why not just delete prompt-predictive n-grams? Feature selection, such as removing n-grams that are strongly correlated with prompt identity, is simpler. It acts only along coordinate axes, however. Character n-grams overlap heavily, so prompt information is spread across many correlated features. The adversary removes arbitrary directions, which covers this case. We will run selection at a matched number of removed dimensions. If it ties, the cross-prompt problem is lexical rather than distributional, and that is itself a finding.

Why not just use an end-to-end adversarial network? On UAS this is exactly DANN, and the published 64.29% WRR figure tells us it works. The case against it rests on small-data stability and cost, as the SLEEP comparison in Deep Neural Baselines for Computational Paralinguistics suggests. That case is only credible if our method matches DANN's accuracy, which is why it is the bar for beating it.

Why not just a nonlinear kernel? In Linear and non linear kernel GMM supervector machines for speaker verification, a KL-divergence-style kernel needed both NAP and model normalization to beat the linear kernel (6.39% vs 6.75% EER). Without normalization it was worse (10.46% vs 8.04%). Text studies such as Kernel Methods for Document Filtering (TREC 2002) found no gain from nonlinear kernels. A nonlinear kernel also does not address domain shift by itself. We will include an RBF SVM as a reference only.

Proposed Experiments

Text testbed. The text testbed is leave-one-prompt-out on the NLI Shared Task 2017 data: 11 first languages, 8 prompts, 13,200 essays. For each fold, seven prompts are labeled source and the held-out prompt's essays are unlabeled target. We report Macro F1 and the gap to in-prompt performance, which measures how much of the reported 20% drop is recovered. We also report the Hindi–Telugu confusion rate, to test whether invariance collapses related classes.

Speech testbed. The speech testbed is UAS isolated digits with 13 healthy and 15 dysarthric speakers. Healthy speech is labeled source and dysarthric speech is unlabeled target, and we report WRR by severity. We also report how many speakers clear the 65% clinical threshold; DANN reaches it for 11 of 15. Supervectors from short utterances are justified by Incorporating Local Acoustic Variability Information into Short Duration Speaker Verification, which found supervector spaces to be more stable than low-dimensional embeddings on short segments.

Decisive checks. Three checks decide the outcome. First, the conditioned adversarial projection must beat closed-form domain NAP on per-class target error, not just on the average. Second, it must remain within seed variance of DANN on UAS. Third, its deleted directions must load on prompt vocabulary or on pathology-affected mixture components, rather than on first-language or digit-discriminative features. A secondary check varies the number of target speakers from 3 to 15, to test whether block structure keeps the projection usable where NAP-style estimation is known to break.