When a text classifier has only a few labeled examples and many unlabeled ones, a standard trick is consistency training: perturb each unlabeled input and penalize the model when its prediction changes. Text makes this hard. Swapping or deleting a token can flip the meaning ("not good" becomes "good"), so the perturbation is either too timid to teach anything or destructive enough to inject label noise. Earlier work avoided the problem in one of two ways. It either perturbed the continuous word embeddings inside a small round ball, or it searched over replacement tokens proposed by a masked language model, which is slow. We propose a third option. Find the worst-case perturbation inside the intermediate hidden states of a Transformer, and constrain it to an ellipsoid shaped by how real sentences of each predicted class actually vary in that layer. The perturbation is then aggressive along directions that natural data already explores and nearly zero along directions it never visits. One budget, measured in units of natural variation, sets the trade-off between how hard and how faithful the augmentation is, and it needs no labels, no decoder back to text, and no token search.
In two sentences: virtual adversarial training on unlabeled text works better when the adversary acts on mid-layer Transformer states and is confined to the covariance ellipsoid of pseudo-labeled classes, instead of an isotropic ball on input embeddings. This converts the "difficult but not too different" principle into a geometric constraint that runs at the cost of ordinary VAT. The prediction is sharpest on short texts, where the isotropic version is known to fail.
Where the gap is
Three lines of prior work each supply one piece and leave out the others. Adversarial Training Methods for Semi-Supervised Text Classification introduced virtual adversarial training (VAT) for text: find the small embedding perturbation that most changes the prediction, then penalize that change. It reached 5.91% error on IMDB. On the very short Rotten Tomatoes snippets, however, its fixed-radius consistency loss overwhelmed the supervised signal and fell below the baseline. That paper motivates our proposal: it shows a single isotropic budget at layer zero is miscalibrated across inputs. Latent Space Virtual Adversarial Training (with its companion Regularization with Latent Space Virtual Adversarial Training) moved the adversary into the latent space of a Glow or VAE model. This cut CIFAR-10 error with 4,000 labels from 16.92% to 9.94%, which is strong evidence that adversaries do better when they follow the structure of the data. The method needs a decoder back to input space, though, and it reports that the blurry VAE decoder hurts. For text, no such differentiable decoder exists. We keep the idea and drop the decoder. Implicit Semantic Data Augmentation (ISDA) supplies the geometric tool. It shows that class-conditional covariance of deep features traces meaningful variations and samples random perturbations from it. ISDA, however, is supervised, random rather than worst-case, and tested only on images.
Two papers constrain the design. Manifold Adversarial Augmentation for Neural Machine Translation perturbs intermediate Transformer layers and beats token-level adversarial augmentation by 0.39 to 1.10 BLEU. That supports working in hidden layers, but the method is supervised and uses static magnitude settings. Consistency Training with Virtual Adversarial Discrete Perturbation (VAT-D) keeps perturbations as real text by searching masked-language-model candidates. It beats BERT by 7.1 points on AG News with 10 labels per class and improves RoBERTa-Large by 8.0 points on ANLI, at roughly 2.5 times the training cost. It is the method to beat, and it also shows that worst-case perturbations matter more than random ones. Finally, What Makes Better Augmentation Strategies? Augment Difficult but Not too Different (DND) states the intensity trade-off explicitly as a difficulty reward against an embedding-similarity reward. Both its difficulty term and its policy training need true labels. Our ellipsoid plays the role of DND's "not too different" term without labels or a learned policy.
Core Claim
Let $h_t^{(l)} \in \mathbb{R}^d$ be the hidden state of token $t$ at Transformer layer $l$ for an unlabeled sentence $x$. Let $\hat c$ be the class the model currently predicts for $x$, and let $\Sigma_{\hat c}$ be a shrunk, low-rank estimate of the covariance of layer-$l$ token states over unlabeled sentences confidently predicted as $\hat c$. We perturb every token state by $\Sigma_{\hat c}^{1/2} u_t$ and choose the unit-free direction $u$ adversarially:
$$ u^\star = \arg\max_{\|u\|_2 \le \epsilon} \; D_{KL}\big(p_\theta(y \mid h^{(l)}) \,\|\, p_\theta(y \mid h^{(l)} + \Sigma_{\hat c}^{1/2} u)\big), $$
Here $p_\theta(y \mid h^{(l)})$ is the classifier output when the remaining layers process the given states, and $\epsilon$ is the single budget, measured in standard deviations of natural within-class variation. As in standard VAT, one power-iteration step approximates $u^\star$, so the method costs one extra forward-backward pass. The training loss is supervised cross-entropy plus the KL term at $u^\star$ on unlabeled data.
The claim is that shaping the adversary by covariance does three things. Hardness comes from the maximization. Fidelity comes from the ellipsoid, which excludes directions real sentences of that class never occupy. The budget adapts automatically across classes and dimensions. A fixed isotropic radius cannot do all three, and it is the isotropic radius, not the use of unlabeled data itself, that caused the Rotten Tomatoes failure.
Research Program
The build is a drop-in consistency loss for RoBERTa-base and RoBERTa-large. Pseudo-classes come from a short supervised warm-up followed by confident predictions on the unlabeled pool. Covariances are re-estimated every few hundred steps with exponential averaging, as ISDA does. To make a 768- or 1024-dimensional estimate stable from limited data, we keep the top-$k$ eigenvectors and add a small isotropic floor. The layer $l$ is chosen once on a held-out task, not per benchmark.
The analysis has three parts. The first measures fidelity without labels, in the "not too different" sense. For each perturbed sentence representation, we compute the distance to its nearest real unlabeled sentence representation at the same layer, at matched KL "difficulty," for isotropic and covariance-shaped adversaries. If the ellipsoid does what we claim, it should produce equal difficulty at smaller off-distribution distance. The second checks whether the method depends on pseudo-labels, by comparing class-conditional covariance against one pooled covariance; if the pooled version matches, the pseudo-labels add nothing. The third studies the robust-overfitting question raised by Overfitting in adversarially robust deep learning, which found late-training robust collapse in vision and that unlabeled data prevents it. We train each method well past convergence and track ANLI and hidden-state attack accuracy over training to see whether text classifiers show the same collapse and whether the ellipsoid delays it.
Baselines
The naive reference is supervised fine-tuning of the same backbone on the labeled examples alone, with no unlabeled data. Every gain is reported relative to it.
The cheapest consistency baseline passes each unlabeled sentence twice with different dropout masks and penalizes disagreement. This is the "minimal augmentation" that SimCSE showed works for sentence embeddings, implemented with its public code (SimCSE repository) adapted to a classification consistency loss.
Embedding-layer VAT follows Adversarial Training Methods for Semi-Supervised Text Classification on the same Transformer, with normalized embeddings and a tuned radius. It is the direct ancestor of our method.
The strongest existing competitors are UDA and VAT-D. Unsupervised Data Augmentation for Consistency Training uses back-translation and word replacement and reports 4.20% IMDB error with 20 labels. VAT-D reports a 7.1-point gain over BERT on AG News with 10 labels per class and an 8.0-point gain on ANLI with RoBERTa-Large; the file gives these as differences, and no absolute VAT-D accuracy was found on Lacuna. No reported result was found on Lacuna for any semi-supervised method's absolute Rotten Tomatoes accuracy.
Two ablations isolate the key ingredient. The first is isotropic hidden-state VAT: same layer, same single power step, but with a round ball instead of the covariance ellipsoid. This separates "hidden layer" from "covariance shape." The second is covariance-random perturbation: ISDA-style random Gaussian samples from the same pseudo-label covariance with no maximization. This separates "ellipsoid" from "adversarial."
For fairness, all methods share the backbone, labeled splits (five seeds), unlabeled pool, number of optimizer steps, and a tuning budget of 20 configurations each. All select on the same small labeled development set, sized to match the training labels, because a large clean validation set is unrealistic in this regime. We also report wall-clock time. Beating VAT-D means matching or exceeding its accuracy on AG News and ANLI at under half its training cost. Beating UDA means a lower IMDB error at 20 labels than our rerun of UDA under the identical protocol, with the reported 4.20% as the external reference point. The headline result requires beating isotropic hidden-state VAT on average across tasks and specifically on Rotten Tomatoes.
Why not just…?
Why not just tune the radius of isotropic hidden-state VAT? This is the real threat. If moving to a mid layer is what matters, the covariance machinery is ornamental. The Manifold Adversarial NMT result is consistent with that reading, and LVAT's gains could likewise come from the latent space rather than its shape. The isotropic ablation, given the same tuning budget, settles it. If it matches our method within seed noise on every task, including Rotten Tomatoes, we will report that the ellipsoid adds nothing and that the contribution reduces to decoder-free hidden-state VAT for semi-supervised text. That result is still new relative to the file, but it is smaller.
Why not just use back-translation and word replacement, as UDA does? UDA's IMDB number is very strong, so this is not dismissed. VAT-D reports, however, that heuristic edits are often too easy and trail it by nearly 3 points on AG News with 10 labels per class. DND reports that fixed heuristics cannot adapt to task-specific sensitivities. Back-translation also requires an external translation system and offline generation. We rerun UDA under the shared protocol. If it wins on long reviews but loses on short snippets and topic labels, that matches the prediction that discrete edits damage short texts.
Why not just sample from the covariance at random, as ISDA does, and skip the adversary? Random sampling is cheaper and inherits ISDA's fidelity argument. VAT-D's finding that random perturbations produce vanishing consistency gradients is the main evidence against it. The covariance-random ablation tests this directly in our setting. If it matches the adversarial version, the "difficult" half of the trade-off is unnecessary once the "not too different" half is enforced.
Why not just use dropout consistency, which is nearly free? SimCSE itself notes that its gains on classification transfer are more modest than on similarity. Dropout noise is also isotropic and undirected, so it should fail on short texts for the same reason embedding VAT does. The dropout baseline measures how much of any gain comes from unlabeled consistency at all.
Why not just learn an intensity policy, as DND does? DND's difficulty reward needs true labels, and its policy search adds substantial cost even after relaxation. Our budget is a single scalar in natural-variation units. Should that scalar turn out to need per-task retuning as sensitive as an isotropic radius, the argument for the ellipsoid as an intensity calibration fails, so we report the sensitivity curve over $\epsilon$ for both shapes.
Proposed Experiments and Decisive Checks
The benchmarks are AG News with 10 labels per class, IMDB with 20 labels, Rotten Tomatoes with matched label counts, and ANLI Rounds 1 to 3 with RoBERTa-Large to test robustness under shift. In the semi-supervised tasks, the remaining training text serves as the unlabeled pool.
The first decisive check is Rotten Tomatoes. Embedding VAT is known to fall below the supervised baseline there. Our method should stay above that baseline, and isotropic hidden-state VAT should not. The second is the flatness of accuracy as $\epsilon$ varies: the covariance-shaped curve should be wider and flatter than the isotropic one, which is the operational meaning of "calibrated intensity." The third is the fidelity diagnostic: at matched consistency loss, perturbed representations should lie closer to real unlabeled sentences. The fourth is cost: wall-clock time within about 1.3 times the naive baseline, versus VAT-D's reported 2.5 times. The fifth is the extended-training learning curves for robust overfitting.
If the first and second checks fail while the overall accuracy gains hold, we conclude that hidden-layer placement, not covariance shape, drives the improvement.
References
- Adversarial Training Methods for Semi-Supervised Text Classification
- Latent Space Virtual Adversarial Training for Supervised and Semi-Supervised Learning
- Regularization with Latent Space Virtual Adversarial Training
- Implicit Semantic Data Augmentation for Deep Networks
- Manifold Adversarial Augmentation for Neural Machine Translation
- Consistency Training with Virtual Adversarial Discrete Perturbation
- What Makes Better Augmentation Strategies? Augment Difficult but Not too Different
- Overfitting in adversarially robust deep learning
- SimCSE: Simple Contrastive Learning of Sentence Embeddings
- Unsupervised Data Augmentation for Consistency Training