With only four labeled examples per class, fine-tuning a 7-billion-parameter language model is unreliable. The standard remedy has three costly stages. First, the model generates synthetic training examples from the few real ones. Second, every generated example goes through a second full prompting pass that checks whether its label is correct. Third, the model is fine-tuned on the cleaned pool. Adding a contrastive loss, which pulls same-class examples together in embedding space and pushes other classes apart, often steadies few-shot training. But it usually requires two forward passes per example, one per "view" of that example. So each fix for instability adds another full pass over the data.
This note proposes that one model and one causal forward pass can do the work of all three stages. When the model finishes generating a synthetic example, we append a short suffix. The hidden state at a designated token in that suffix then serves as the example's embedding at almost no extra cost. That embedding replaces the second-pass label check with a nearest-prototype test against the real examples. During fine-tuning, we pack each real seed and its synthetic descendant into a single sequence, so the model sees two related views in one pass. We use them for a contrastive loss that knows which real example each synthetic one came from. The claim is specific: at equal or lower total compute (generation plus filtering plus training), this pipeline matches or beats generate-filter-train with cross-entropy, and it lowers run-to-run variance across draws of the few labeled examples.
Core Claim and Its Place in Prior Work
The proposal directly extends Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data. That work has Vicuna-7B generate examples from 4 seeds per class, discard label-inconsistent ones by re-classifying each with in-context learning (prompting with labeled demonstrations, no weight updates), and then fine-tune itself with low-rank adapters (LoRA, a small trainable weight update added to frozen layers) and cross-entropy. It motivates this proposal in two ways. Its ablation shows the filter matters: accuracy on TREC rose from 0.68 unfiltered to 0.79 filtered. Its filter is also a full extra autoregressive pass per candidate.
The proposal borrows its mechanism from CSE-SFP. CSE-SFP arranges a decoder input so that the causal attention mask yields two representations of one sentence in a single pass. It trains unsupervised sentence embeddings with QLoRA, running 43% faster (280 to 159 minutes on Llama-3 8B) and using about 8GB less memory. CSE-SFP never touches few-shot supervised tasks or synthetic data. We change what the two positions represent: a real seed and a model-written example derived from it, rather than two readings of the same sentence.
Two further papers constrain the design. Few-shot Text Classification with Dual Contrastive Consistency (FTCC) shows supervised contrastive training helps at 10 shots, reaching 86.43% on SST-2 against 67.76% for plain BERT fine-tuning. It pays for this with back-translation and multi-branch passes, and it uses existing unlabeled text rather than generated text. Nearest Neighbour Few-Shot Learning for Cross-lingual Classification (NNFS) shows that mean-subtracted cosine distance to class prototypes over frozen features beats few-shot fine-tuning on XNLI and PAWS-X at 38 to 40 times less cost. That result is why we expect a geometric filter to be competitive with a prompting filter.
The closest joint treatment, Contrastive Decoding for Synthetic Data Generation in Low-Resource Language Modeling, is different in kind. It applies contrast during decoding, subtracting an amateur model's scores from an expert's. That doubles generation cost, and the student is trained with ordinary language modeling. Its finding that more than 30–40% synthetic data hurts perplexity concerns pretraining. We treat it as a warning to sweep the real-to-synthetic ratio, not as a result about classification.
Research Program
We would build the pipeline on Vicuna-7B, matching the COLING setup, with a second decoder backbone (Llama-3 8B or Mistral-7B) for robustness. The pipeline has three pieces.
The first piece is generation that also produces an embedding. For each seed with label $y$, we prompt the model to write a new example of class $y$. After generation ends, we force-decode a fixed suffix of a few tokens, for example "This text in one word means: [R]". Because the key-value cache already exists, the hidden state at [R] costs only those few tokens.
The second piece is the filter. We compute class prototypes from the [R] states of the real seeds, embedded with the same suffix, using NNFS-style mean subtraction. A candidate is kept if its nearest prototype matches its intended label with margin $\delta$. We tune $\delta$ only by leave-one-out on the seeds, never on test data. The per-candidate filtering cost drops from one full prompted pass to a few decoded tokens plus a dot product.
The third piece is training with lineage-aware contrast. Each training sequence packs a seed $s$ and one of its kept descendants $g$ as "[instruction] $s$ [R$_1$] [bridge] $g$ [R$_2$]". R$_1$ can attend only to the seed; R$_2$ can attend to both. The loss is cross-entropy on the label from R$_2$ plus a supervised contrastive term over the in-batch [R] states $z$:
$$\mathcal{L} = \mathcal{L}_{\text{CE}} - \lambda \sum_{i} \frac{1}{|P(i)|} \sum_{p \in P(i)} \log \frac{\exp(z_i \cdot z_p / \tau)}{\sum_{a \neq i} \exp(z_i \cdot z_a / \tau)}.$$
Here $P(i)$ is the set of same-label representations for anchor $i$, the seed–descendant pair gets an additional up-weight $w$, $\tau$ is a temperature, and $\lambda$ weights the contrastive term. Only LoRA parameters are trained.
After one round, the fine-tuned [R] space can re-filter a fresh generation batch. We test this optional second round separately, because it risks the model confirming its own errors.
We also analyze three things: agreement between the geometric filter and the prompting filter; how rejection rates vary by class; and whether contrastive training tightens class clusters in a way that predicts lower seed variance.
Baselines
The naive reference points are in-context learning with the same 4 seeds per class and LoRA with cross-entropy on the seeds alone. On TREC at 4 shots, the COLING paper reports 0.60 accuracy for in-context learning and describes seed-only LoRA as often no better than zero-shot. These set the floor.
The strongest direct competitor is the full generate-filter-train pipeline of Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data. It reports 0.84 accuracy on 4-shot TREC with Vicuna-7B. We will rerun it rather than copy the number, using the same generation prompts, the same number of raw candidates per seed, the same LoRA rank, the same training steps, and the same hyperparameter search budget. Every run uses the same five random seed-set draws. Beating it means one of two outcomes. Either accuracy is equal within the seed standard deviation at clearly lower total GPU-hours, or accuracy is higher at equal GPU-hours. In both cases the across-draw standard deviation must also be lower. The file reports no GPU-hour figure for this pipeline's filtering stage, so we will measure it ourselves.
The contrastive competitor is FTCC-style supervised contrast with two separate passes per example, run on the same filtered synthetic pool. It isolates what single-pass packing contributes. The reference number is FTCC's 86.43% on SST-2 at 10 shots (BERT). We will also report our method at 10 shots on SST-2 for context, but not as a head-to-head comparison.
For the wider suite, the strongest reported number is the 79.46% 16-shot average of the retrieval-augmented objective in Retrieval-Augmented Few-shot Text Classification, against a 73.87% vanilla baseline.
Three ablations isolate the key ingredients. The first swaps the geometric filter for the prompting filter while keeping contrastive training. The second drops the contrastive term while keeping the geometric filter. The third sets the lineage weight $w=1$, leaving only label positives.
Why not just…?
Why not just filter with a cheap off-the-shelf sentence encoder instead of the generator's own hidden state? This is a real threat. An external encoder plus nearest-prototype filtering would also remove the prompting pass. We will run it with NNFS's mean subtraction and the same margin search. If it agrees with the prompting filter as often as ours does, the "same pass" argument reduces to a small convenience. The remaining contribution would then rest on the training side. Our expectation is that the generator's own representation separates task-specific classes better than a generic encoder, but that is what the experiment decides.
Why not just skip fine-tuning and run nearest-neighbor inference over real plus synthetic prototypes? NNFS shows frozen metric inference beats few-shot fine-tuning at a fraction of the cost. Adding synthetic prototypes might close most of the gap to 0.84 at almost no cost. We include this as a baseline. If it comes within seed noise of our method on TREC and SST-2, we will report that the training stage is not worth its cost for single-label classification.
Why not just keep cross-entropy and add more synthetic data? The COLING authors report a diversity ceiling as synthetic data scales, and SynAlign documents a "monochrome style" gap in LLM generations. Both suggest volume alone saturates. We sweep candidates per seed (4× to 32×) for both cross-entropy and contrastive training. The contrastive loss should matter most where cross-entropy plateaus. If the curves do not separate, it does not help.
Why not just use standard two-pass contrastive training, since compute is cheap at 7B with LoRA? The FTCC-style baseline answers this directly. If packing gives equal accuracy, we still report its measured wall-clock and memory savings against CSE-SFP's reported 43%. If packing hurts accuracy, for example because R$_2$ attending to the seed leaks information, the packed design is rejected in favor of two passes.
Proposed Experiments and Decisive Checks
The benchmarks are TREC and SST-2 at 4 shots per class, following the COLING protocol, and the 10-dataset suite (SST-2, MR, CR, TREC, QQP, QNLI, MNLI, SNLI, RES, LAP) at 8 and 16 shots, following the retrieval-augmented paper. We report accuracy with mean and standard deviation over five seed-set draws. Compute is reported in GPU-hours, split into generation, filtering, and training.
Four checks decide the outcome. First, filter fidelity: on a candidate pool labeled by hand plus the prompting filter, the geometric filter's precision on correctly labeled examples should come within a few points of the prompting filter's, at a small fraction of its cost. Second, noise robustness: deliberately flipping 10–30% of synthetic labels should hurt our method less than the cross-entropy pipeline. Third, variance: the across-draw standard deviation should drop against the rerun COLING pipeline on at least most of the datasets. Fourth, the ratio sweep: we vary the synthetic-to-real mix and check whether gains hold beyond the regime where Contrastive Decoding for Synthetic Data Generation in Low-Resource Language Modeling saw degradation.
The proposal fails if both of the following hold: the external-encoder filter plus cross-entropy matches it, and nearest-neighbor inference over synthetic prototypes comes within noise. We would report that negative result plainly.
References
- Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data
- CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass
- Few-shot Text Classification with Dual Contrastive Consistency
- Nearest Neighbour Few-Shot Learning for Cross-lingual Classification
- Contrastive Decoding for Synthetic Data Generation in Low-Resource Language Modeling
- Retrieval-Augmented Few-shot Text Classification
- Few-shot LLM Synthetic Data with Distribution Matching