Spend the Verifier Where the Student Leaks: Budgeted Hard-Negative Synthesis for Out-of-Scope Detection

Intent classifiers in dialogue systems are trained as if every user message belongs to one of a fixed list of supported intents. When a message falls outside that list, the classifier still assigns it to some intent, often with high confidence. The hardest cases are "near" out-of-scope requests. These reuse the vocabulary of a supported intent but ask for something the system cannot do, such as "cancel my mortgage" sent to a bot that only handles card cancellations. Large language models (LLMs) can write such hard negatives on demand. The pipelines that do this reliably, however, are costly, and most of the cost goes to checking the output rather than producing it. Every generated candidate is verified by further LLM calls, sometimes by debate panels or human reviewers, because writing near the boundary produces many mislabeled examples.

We propose a closed-loop recipe built on one observation. Most hard negatives an LLM writes are ones the small deployed classifier (the "student") already rejects, and verifying those is largely wasted money. A cheap generator should produce a large pool of candidate near-out-of-scope utterances. The student scores them, and only the candidates it wrongly accepts go to an expensive verifier. These "leaks" are both the examples that teach the student something and the examples most likely to be mislabeled, so this is exactly where verification is worth paying for. Each round's confirmed leaks then steer the next round of generation toward the intents that leak most. The claim we want to test is that this matches or beats existing hard-negative pipelines on human-written out-of-scope test data while using a small fraction of their verification calls.

Core Claim

The proposal treats out-of-scope detection as the application, LLM augmentation as the mechanism, and generation overhead as the budget constraint. Its testable claim is specific. At a fixed budget of verifier tokens, sending verification only to student-accepted candidates, and shifting generation toward high-leak intents, yields lower false-positive rates on real near-out-of-scope queries than uniform, open-loop hard-negative generation. It also reaches the open-loop pipeline's accuracy with several times fewer verifier calls. A secondary claim is that the same loop recovers useful in-scope examples for free. When the verifier overrules a candidate because it is actually in-scope, that utterance becomes a hard positive for the intent it was mistaken for.

Relation to Prior Work

The closest work is Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification. It mines frequent keywords per intent, asks ChatGPT for utterances that contain those keywords but fall outside the intent, and verifies each one with two LLM checks plus a human pass. It produced 3,732 hard negatives across CLINC150, Banking77, ATIS, Snips and HWU64. This paper both motivates and constrains our design. It shows that keyword-targeted hard negatives help, reporting AUROC of 0.996 on Banking77 when trained on in-scope plus hard-negative data. It also shows that about 1.2% mislabels survive automated verification. Its generation is open-loop: it never asks whether the student already handles a candidate, and it does not measure token cost. We keep its keyword-conditioned prompting as a starting point and change where generation and verification effort are spent.

Two other LLM-augmentation papers supply pieces of the method and point to what is missing. PromptMix prompts GPT-3.5 for examples that blend two classes and then relabels every one with a second LLM pass before distilling into BERT-sized students. It reports that relabeling improves accuracy by up to 13.5% and reaches 80.1% at 2-shot on Banking77. Its boundaries lie between known classes only, it never produces out-of-scope data, and it doubles query cost uniformly. Out-of-Distribution Detection using Synthetic Data Generation uses Llama-3 70B to write near and far outliers from static templates for document-level tasks. It reports near-OOD FPR95 of about 10% against about 61% for the best baseline, and notes a remaining gap to detectors trained on real outliers. Neither paper uses student feedback.

BARRED constrains our verifier design. Removing its multi-LLM debate verification cost 27% accuracy on human-annotated data, and single-model self-critique was worse than no verification at all. We therefore never let the generator verify its own output. Our verifier is a separate, stronger model, and we spend it sparingly rather than everywhere.

Several papers use the student's uncertainty or geometry without generating text from it. UDRIL computes a DistilBERT uncertainty score but uses it at run time, sending uncertain queries to a fine-tuned Llama 3.1-8B. Its reported gain is a 2–3% relative improvement on HINT3. Navigating the Unknown likewise keeps a frontier LLM in the live loop as a verifier for DistilRoBERTa. Both pay LLM cost on every uncertain user query, forever. We pay it once, offline, and deploy only the encoder.

EliDecide and A Closer Look at Few-Shot Out-of-Distribution Intent Detection tighten the student's boundaries using outliers made by mixing hidden vectors. These outliers cost nothing to create but never correspond to real sentences. GOLD filters retrieved pseudo-outliers with an ensemble of detectors, which is close in spirit to letting a cheap model decide what is informative. GOLD filters retrieved text to throw away likely mistakes, whereas we use the student to route generated text toward expensive verification.

Research Program

The loop has four parts.

The first part is the student. It is a DistilBERT or BERT-base encoder fine-tuned on in-scope data, and it produces an out-of-scope score $s(x)$ for each utterance $x$. We test two scoring heads. One is Mahalanobis distance to class centroids, following Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection. The other is a $(K+1)$-way head with an extra "none of the above" class. A rejection threshold $\tau$ is set so that 95% of held-out in-scope data is accepted.

The second part is a cheap generator. This is an open 7–8B instruction model that writes candidates by three operations, each starting from an in-scope seed. Keyword-preserving action swaps keep the entities but change the requested action, adapting the keyword prompts of the hard-negative paper. Scope shifts keep the action but move it to an unsupported object or domain. Free near-domain requests cover the rest of the space. Generation is cheap per token, so we over-generate.

The third part is student routing. A candidate is sent to the verifier only if the student accepts it as some intent $k$, that is, $s(x) < \tau$. The verifier is a separate frontier model prompted with the full intent inventory, using the two-question protocol of the hard-negative paper. Confirmed out-of-scope leaks become training negatives. Candidates the verifier calls in-scope become positives for their verified intent. Rejected-by-student candidates are not verified. We add only a small, randomly verified fraction of them to the training set, so that rejection behaviour stays anchored to some confirmed negatives as well.

The fourth part is allocation across rounds. Each round splits the next generation budget over intents in proportion to their measured leak rate:

$$n_k \propto \epsilon + \frac{\#\{\text{confirmed out-of-scope candidates accepted as intent } k\}}{\#\{\text{candidates generated from intent } k\}}$$

Here $n_k$ is the number of candidates generated from intent $k$ next round, and $\epsilon$ is a small floor so that no intent is starved. Confirmed leaks from intent $k$ are also inserted as in-context examples in the next prompt for that intent. The student is retrained after each round. We run a fixed number of rounds until the total budget is spent.

Every run logs generator tokens, verifier tokens, human-check minutes and wall-clock time. Results are always reported as accuracy against cost curves, not single points. The analytic question running through all experiments is how often student-accepted candidates are mislabeled compared with student-rejected ones. If our premise is right, mislabels concentrate among the accepted candidates. That would make verifying only those both cheaper and more effective at removing noise.

Baselines

The naive reference point is the student trained on in-scope data only, with maximum-softmax rejection. On CLINC150, An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction reports 96.9% in-scope accuracy for BERT but out-of-scope recall of only 40–60%. This marks the overconfidence any method must fix.

The strongest data-free competitor is the same encoder with Mahalanobis scoring. Revisiting Mahalanobis Distance reports 98.4% AUROC on CLINC150. Bounded-region heads are also run: ADB, which reports 88.53% F1 at 75% known intents on CLINC150 in Deep Open Intent Classification with Adaptive Decision Boundary, and EliDecide, which reports 84.48 F1 at 25% known-class ratio against ADB's 79.18. These are strong because they cost nothing to train beyond fine-tuning. Our method must beat them specifically on near-out-of-scope test data, where the file indicates distance methods struggle.

The strongest existing synthetic-data method is the open-loop hard-negative pipeline of Generating Hard-Negative Out-of-Scope Data with ChatGPT, with its reported 0.996 AUROC on Banking77. That number was measured against its own generated hard negatives, which is why our primary test sets are human-written. We re-implement this pipeline with the same generator, verifier and student as our method. We give it the same total token budget, with every candidate verified twice as in the original, and the same number of hyperparameter trials. Its human pass is replaced by the same small audit we apply to our own data. No reported cost figure for this pipeline was found on Lacuna, so we measure it ourselves.

We also run a template-based near-OOD generator in the style of Out-of-Distribution Detection using Synthetic Data Generation under the same budget. As a cost reference rather than a competitor, we report a runtime LLM verifier in the style of Navigating the Unknown, with its per-query inference cost included.

Two ablations isolate the key ingredient. The first, random routing, sends the same number of verifier calls to a random subset of candidates instead of the student-accepted ones. The second, uniform allocation, keeps student routing but drops leak-proportional budgets and leak exemplars. An upper bound verifies every candidate. Beating the open-loop pipeline means a statistically reliable FPR95 reduction on human-written near-out-of-scope data at equal total cost. Alternatively, it means matching that pipeline's accuracy at no more than one third of its verifier tokens. If random routing performs as well as student routing, the core claim fails.

Why not just…?

Why not just generate more hard negatives open-loop with the same money? This is the most serious threat. LLM-written negatives may be diverse enough that targeting adds little. Two pieces of evidence argue against it. Beyond Quantity reports early diminishing returns from untargeted synthetic scaling, and SynOOD found gains peaking and then declining past 50,000 synthetic samples. But these come from other domains. The budget-matched comparison and the random-routing ablation settle the question directly for intent detection. If the curves overlap, we will report that student guidance does not pay off.

Why not just use free latent-space mixup outliers? Mixed hidden vectors, as used by EliDecide, cost no LLM calls. However, they lie between known classes rather than near them in surface form. They therefore cannot represent "same keywords, different request," which is the near-out-of-scope failure the hard-negative paper documents. We test this by adding our text negatives on top of EliDecide's contraction loss. If mixup alone matches the combination on keyword-sharing test queries, the generation step is unnecessary.

Why not just keep an LLM in the loop at inference, as UDRIL does? Runtime routing works, as UDRIL shows. But it charges an 8B-or-larger model on every uncertain query for the life of the system, and it depends on a hand-tuned routing threshold. We report total cost over a projected query volume. That makes the crossover point explicit: the point beyond which one-time offline synthesis is cheaper than routing.

Why not just mine real unlabeled logs instead of generating? GOLD-style retrieval from GOLD is cheap when large in-domain corpora exist. Many new deployments lack such corpora, and retrieval does not specifically produce keyword-sharing negatives. Where real logs exist (HINT3), we run student-routed verification over retrieved candidates as a variant. This also tells us whether routing helps independently of generation.

Proposed Experiments

The in-scope training sets are CLINC150, Banking77, HWU64, ATIS and Snips. We use reduced-shot variants, including 10 examples per intent as in Efficient Intent Detection with Dual Sentence Encoders, because that is where synthetic data should matter most. We also use known-class-ratio splits of 25%, 50% and 75% on CLINC150, Banking77 and StackOverflow, so results compare directly with ADB and EliDecide.

Test-time out-of-scope data is mostly human-written. It includes the CLINC150 out-of-scope set and held-out intents from the known-class-ratio splits, which are near-domain by construction. It also includes HINT3 real chatbot logs and ROSTD. The released hard-negative set from the COLING paper is reported separately as a synthetic-distribution check, if it is available. Our generator never sees any test utterance or held-out intent name.

Metrics are AUROC, FPR95, out-of-scope recall at fixed in-scope accuracy, and macro-F1 over known plus open classes. Each is plotted against verifier tokens, generator tokens and total cost.

Three results would be decisive. The first is whether student routing beats random routing at equal verifier budget on near-out-of-scope FPR95; this is the core claim. The second is whether the full loop reaches the open-loop pipeline's accuracy at a third or less of its verifier cost. The third is whether, in a human audit of 500 candidates per stratum, mislabel rates are measurably higher among student-accepted candidates than among student-rejected ones. That would confirm that routing concentrates verification where noise lives.

A negative result on the first check, or no gain over Mahalanobis on held-out-intent splits, would show that student feedback is not the missing ingredient. Either outcome usefully narrows what synthetic hard negatives are worth paying for.