Auditing Prompt Sensitivity on a Budget for On-Device Self-Training

Small language models running on phones and embedded boards meet data unlike their training data, such as new product domains, new user vocabularies, or new intent phrasings. One way to adapt them without human labels is to let them label incoming text themselves and fine-tune on those guesses. That approach is called self-training, and the guesses are called pseudo-labels. Its known failure is that wrong guesses get reinforced. The usual defense is to keep only the guesses the model is confident about, but confidence is exactly what breaks under distribution shift, and it breaks further when the model is compressed to 4 bits. A different signal is available. When a model is unsure, its answer tends to flip under meaningless rewordings of the prompt: an extra newline, reordered options, a paraphrased instruction. Checking for such flips is a better error detector than confidence, but it costs several forward passes per input, which an edge device cannot afford on every sample.

We propose to run the expensive multi-prompt check on only a small, deliberately chosen fraction of the incoming stream. Those audits train a tiny probe that predicts, from a single normal forward pass, whether the model's answer would survive rewording. The probe's prediction then decides which pseudo-labels a 4-bit model is allowed to learn from. The claim is that this recovers most of the filtering quality of checking every sample at close to single-pass cost. It should also stay reliable after quantization and after the model has started adapting, which is where confidence-based filters fail.

Core Claim

The claim has three parts, each of which could fail. First, the correlation between prompt sensitivity and error survives 4-bit quantization and may even strengthen. ProSA ties sensitivity to flat output distributions, and quantization plausibly flattens them. Second, a linear probe on the canonical-prompt hidden state can predict multi-prompt disagreement well enough to replace most audits, provided it is refreshed by a small online audit budget. Third, pseudo-labels filtered this way produce better continual self-training on a quantized model than confidence- or entropy-based filters at matched energy.

What Prior Work Establishes and Leaves Open

Several papers motivate the signal. On the Relation between Sensitivity and Accuracy in In-Context Learning (SENSEL) measures answer flips across instruction paraphrases and demonstration orderings in GPT-J-6B and GPT-Neo-2.7B. It finds correlations with correctness as strong as −0.40 and uses the signal to abstain, gaining +4.1 points of area under the F1-coverage curve over maximum softmax probability on GPT-J-6B. It never updates weights, it pays K forward passes per query, and it studies only unquantized models. Brittlebench shows that the signal is large: prompt perturbations account for up to 50% of performance variance in open-weight models. Its evaluation is also cloud-precision only.

Two papers supply the pieces we reuse. Less is More: Pseudo-Label Filtering for Continual Test-Time Adaptation (PLF) contributes class-specific self-adaptive thresholds and a class-balance regularizer. We reuse both, but apply them to a sensitivity score rather than softmax confidence. SAML shows that low-rank adapters (LoRA, small trainable matrices added beside frozen weights) can adapt a 4-bit NF4 base and even surpass the full-precision model. That is supervised speaker adaptation; we use the same parameter-efficient update, but label-free.

Other papers constrain the design. Prompt Ensemble Self-training (PEST) is the closest existing combination of prompt variation and pseudo-labeling. It averages over LLM-generated prompts, image views, and time to produce better labels for CLIP. Its cost is full ensembling on every sample, offline, which is the cost we try to amortize. We also use prompt disagreement as a reliability filter rather than as an averaged prediction. On-demand Test-time Adaptation for Edge Devices (OD-TTA) shows that selective, on-demand updating beats always-on adaptation on a Jetson Orin Nano: 78.0% vs TENT's 10.1% on CIFAR-10-C at batch size 1, with up to 47.1% energy savings. It is bound to batch normalization and triggers on entropy. A Decision-Theoretic View of Test-Time Training warns that too many update steps on noisy targets cause accuracy cliffs, which argues for bounded, filtered updates. Its Jacobian-based selection is too expensive for our setting.

Research Program

The model is a small open-weight instruction-tuned language model from the families Brittlebench evaluates (Llama, Qwen). It is quantized to 4-bit NF4, with frozen weights and LoRA adapters as the only trainable parameters. It classifies text by scoring label tokens under a canonical prompt. A pool of K semantics-preserving variants is built from the perturbation types in Brittlebench and ProSA: whitespace and newline edits, option reordering, label-symbol swaps, and instruction paraphrases.

For each incoming input, the device runs the canonical prompt once. It reads the final-token hidden state $h(x)$ and a linear probe outputs $\hat d(x)$, the predicted probability that the answer would flip under at least one variant. With probability set by an audit budget $\rho$, for example 5–10% of inputs, the device runs all K variants. Audits are sampled preferentially where $\hat d$ is near 0.5, since those are the inputs where the probe is least sure. Each audit yields the true disagreement, used to update the probe, and a majority-vote label. Non-audited inputs are accepted as pseudo-labels when $\hat d(x)$ falls below a PLF-style class-specific adaptive threshold. Each accepted input is weighted by $1-\hat d(x)$ in the cross-entropy loss on the LoRA parameters. Updates run on demand, when the running mean of $\hat d$ rises, in the spirit of OD-TTA, and for a bounded number of steps.

The main risk is that the signal gets gamed. If self-training makes the model consistent on its prompts without making it correct, sensitivity stops predicting error. We guard against this in two ways. Training uses only the canonical prompt. Auditing uses a rotating subset of variants that is never trained on. We also track the sensitivity–correctness correlation over the stream on a small labeled monitoring split, which is used only for evaluation.

A preliminary study on Gap 1 comes first. We measure the sensitivity–error correlation and the probe's predictive quality at bf16, int8, and NF4, with no adaptation. If the correlation collapses under 4-bit quantization, the adaptation program is abandoned for that precision and the negative result is reported on its own.

Baselines

The naive reference is the source-only quantized model run with the canonical prompt and no adaptation. Every other method must beat it, and it shows whether self-training helps at all under 4-bit weights.

The strongest cheap competitor is a PLF-style filter transplanted to the language model. It uses class-specific self-adaptive thresholds on maximum softmax probability, plus class-prior alignment, with the same LoRA updates and the same on-demand trigger. It is the strongest because it has the same single-pass cost and already handles non-stationary shift. PLF's published numbers are on image corruptions, and there is no reported result on Lacuna for PLF on text or on quantized language models, so we will run it ourselves. An entropy-trigger variant after OD-TTA and a fixed 0.9-confidence filter are included as weaker cheap baselines.

The strongest expensive competitor is full SENSEL-filtered self-training, which runs all K variants on every input and keeps majority-vote labels with low disagreement. A PEST-style variant averages the K prompts into the pseudo-label. The only published number for this family is SENSEL's +4.1 AUC over maximum softmax probability on GPT-J-6B for abstention. There is no reported result on Lacuna for sensitivity-filtered self-training. This baseline is the accuracy ceiling our method tries to approach at roughly 1/K of its extra cost.

As an external reference on Amazon Reviews, PL-Mix reports 93.84% average accuracy over 12 cross-domain pairs with RoBERTa-base. That setting uses labeled source data and full-precision offline training, so it is an upper reference from a more favorable regime, not a like-for-like competitor.

Two ablations isolate the key ingredient. The first freezes the probe after offline training, with no audits, and tests whether online refreshing matters once the model adapts. The second replaces uncertainty-directed audits with random audits at the same budget.

All methods share the model, adapter rank, stream orderings, update-step cap, and a hyperparameter search of equal size tuned on a held-out source-domain stream. Budgets are matched in measured energy on device, not only in pass counts. Our method counts as beating the field if two conditions hold across at least five stream orderings: it exceeds the PLF-style filter with non-overlapping confidence intervals at no more than 1.2× the canonical-pass energy, and it closes most of the gap to full-K filtering, which we pre-register as at least two thirds.

Why not just…?

Why not just use confidence thresholds, as PLF does? This is the real threat. SENSEL's advantage over maximum softmax probability is only +4.1 AUC in full precision, so a well-tuned adaptive threshold may be close. Our bet is that the gap widens precisely under 4-bit quantization and during adaptation, when softmax confidence inflates on the model's own mistakes. The PLF-style baseline, run on identical streams and energy, settles this. If it matches us, we report that sensitivity adds nothing on-device.

Why not just run the full multi-prompt check overnight while the device charges? For tasks that tolerate day-scale adaptation, this may be good enough, and the full-K baseline measures exactly its accuracy. The probe matters only when the model must adapt within a session, or when energy and thermal budgets bind. We will report both regimes and say plainly where the cheaper method is unnecessary.

Why not just make the model prompt-invariant before deployment, with consistency training? Invariance removes brittleness on the source distribution, but it cannot remove the uncertainty that shift induces. It may also destroy the very signal that flags errors on new domains. We test this by applying our pipeline to a consistency-trained base and checking whether the sensitivity–error correlation on shifted domains survives.

Why not just use Monte Carlo dropout or input augmentation for uncertainty? Both are also multi-pass, and they have no direct text analogue to reordering options or rephrasing instructions. We include a K-sample dropout filter at matched budget as a sanity comparison.

Proposed Experiments

The continual-shift stream uses Amazon Reviews domains in sequence, ordered as in PL-Mix's category pairs, with ten orderings in the style of PLF. The ten-task classification suite and AG News from SENSEL test probe validity. The SmartLights and Music intent datasets from Spoken Language Understanding on the Edge are run on text transcripts, with the Music popularity tiers serving as a natural long-tail shift. Metrics are accuracy or intent F1 along the stream, pseudo-label precision and acceptance rate, the probe's AUROC for predicting disagreement, and the running sensitivity–correctness correlation. On a Jetson Orin Nano we measure energy per input, latency, and peak memory.

Three results would be decisive. The first is the precision study: whether the sensitivity–error correlation at NF4 is at least as strong as at bf16. The second is the anti-gaming check: whether that correlation stays stable through adaptation when held-out variants are used, and whether it decays when training variants leak into the audit pool. The third is the cost–accuracy curve as $\rho$ sweeps from 0 to 1. The claim holds if the curve reaches most of full-K accuracy by $\rho \le 0.1$ and sits above the PLF-style filter at every matched energy.