Fork Late, Drop Tokens: Cheaper Contrastive Sentence Embeddings

Contrastive learning trains a sentence encoder by showing it two slightly different "views" of the same sentence and teaching it to map them close together while pushing other sentences away. For images, making a second view is easy: crop or flip the picture. For text it is hard. Deleting or swapping a word can break the grammar, and removing a single "not" can flip the meaning. The dominant workaround is SimCSE, which feeds the same sentence through the encoder twice and lets random dropout supply the difference. That doubles the training compute, and the two views barely differ. This proposal takes a different route. It runs each sentence once through the lower layers of a bidirectional encoder, then forks into two branches for only the upper layers. In each branch, a different random subset of token positions is hidden from attention.

The bet is that one design choice fixes both problems at once. By the middle of a Transformer, each position already carries information from its neighbors: a negation has spread into the verb and object it modifies. Hiding a position at that depth is therefore a much gentler operation than deleting the word from the input. Forking late also means the shared lower layers are computed once instead of twice, and the dropped tokens make the upper branches shorter. The fork depth becomes a single knob that trades cost against view diversity. We expect a middle setting to match or beat SimCSE at roughly two-thirds of its compute, without the meaning flips that input-level word deletion causes.

Core Claim

Token dropping is destructive when applied to raw text but safe once tokens are contextualized. Applying it only in the upper layers of a shared-trunk encoder gives contrastive views that differ more than dropout views, keep meaning better than input deletion, and cost less than two full forward passes.

For an encoder with $L$ layers, a fork after layer $k$, and a fraction $p$ of positions hidden in each branch, the cost of the training forward pass relative to SimCSE's two full passes is approximately

$$ C(k,p) = \frac{k + 2(L-k)(1-p)}{2L}. $$

This follows from counting layer-token operations and ignoring the quadratic attention term, which only makes shorter branches cheaper still. For BERT-base ($L=12$) with $k=6$ and $p=0.2$, $C = 0.65$. The backward pass scales similarly, and activation memory for the shared lower layers is stored once rather than twice.

Relation to Prior Work

Three papers motivate the gap.

  • SimCSE established dropout as a minimal augmentation, run as two full passes of a bidirectional encoder. We keep its loss and data but replace the second full pass.
  • DiffAug shows dropout views are too similar to give a strong training signal. It fixes this with learned continuous prefixes, at about 1.8x SimCSE's training time and with a sensitive two-stage warmup. We aim for harder views with no extra parameters and no warmup.
  • CSE-SFP removes the second pass by exploiting the one-directional attention of decoder language models. It reports a 43% speedup and about 8GB of saved memory on an 8B model. That trick does not transfer to BERT-style encoders, where every token sees every other token. Our proposal is an encoder-side answer to the same overhead problem.

Two papers constrain the design.

These results rule out simply deleting input words to make the second view cheap, and they supply the failure we test for.

The closest method is EfficientCL. It avoids sampling a second text span by applying cutoff (zeroing spans of the hidden sequence) and principal-component jittering to one anchor's representations, and it reports using 70% of DeCLUTR's memory. We differ from it in three ways:

  • We perturb only above a chosen depth, where tokens are contextualized.
  • We hide positions from attention rather than zeroing embedding values that later layers still read.
  • We share the lower-layer computation between the views.

EfficientCL also reports a drop on CoLA, which we treat as a warning that blind hidden-state corruption can erase fine-grained syntax. We measure the same risk.

SDA supplies a tool rather than a competitor. Its single-negation operation produces sentences that are syntactically identical to the original but opposite in meaning. We reuse it to build a negation-sensitivity probe.

Research Program

We implement the forked encoder in a standard BERT or RoBERTa codebase. Layers 1 to $k$ run once per sentence with one dropout sample. The layer-$k$ hidden states are then copied into two branches. Each branch drops its own random $p$ fraction of non-special token positions and runs layers $k+1$ to $L$ with independent dropout. The two sentence vectors, taken at the first-token position, form the positive pair for the usual in-batch InfoNCE loss. Gradients from both branches flow into the shared trunk.

At inference nothing is dropped and the encoder is used unchanged. The method therefore adds no parameters and no deployment cost.

The analysis centers on sweeping $k$ from 0 to $L-1$ and $p$ from 0 to 0.4. Each setting is recorded on three axes:

  • View divergence: mean cosine distance between the two branch outputs.
  • Meaning preservation: the negation probe described below.
  • Downstream quality and cost: STS scores, measured wall-clock time, and peak memory.

The setting $k=0$ is input-level word deletion. Setting $p=0$ with late $k$ is dropout-only forking. The sweep therefore contains the obvious alternatives as special cases, which makes the comparison internal and fair.

We add one targeted variant to test whether contextualization really makes random dropping safe. It exempts negation words and the highest-attention tokens from being dropped, following the attention-guided selection idea of ATMIX. If this exemption helps at small $k$ but not at larger $k$, that is direct evidence for the core claim.

Baselines

Unsupervised SimCSE (naive reference). This is two full dropout passes of BERT-base and RoBERTa-base on the standard 1M English Wikipedia sentences. It is the reference every method in this area reports against. We rerun it with its published hyperparameters on our hardware so that time and memory are measured identically. The file lists SimCSE's reported unsupervised STS average as 76.3%, and CompCSE reports SimCSE BERT-base at 76.25.

Compute-matched SimCSE. This second naive reference gives SimCSE the same total floating-point budget as our $k=6, p=0.2$ setting, realized as fewer training steps. This guards against the possibility that SimCSE simply converges early and our savings are free.

CompCSE (strongest comparable competitor). CompCSE contrasts a sentence with its composed constituents. It is cheap, reported to train in under 15 minutes on one A100, and it reports 78.18 for BERT-base (78.97 with its tuned negative sampling). It uses the same unsupervised data and backbone, so it is the most direct accuracy target.

SDA. SDA reports an overall STS improvement from 78.49% to 79.60%. We cite this as reported, since the backbone behind that figure is not stated in our file.

DiffAug. DiffAug reports 82.42 on STS-B against 81.57 for its SimCSE reference. Its warmup uses labeled NLI data, so we treat it as an upper reference for quality and a lower reference for efficiency, not as a like-for-like comparison.

EfficientCL-style cutoff. We reimplement cutoff applied to hidden states under the SimCSE objective, with a single encoder pass. This is the closest single-pass encoder competitor. The file reports no STS number for it, since EfficientCL was evaluated on GLUE, where the file lists an 85.34 dev average for EfficientCL against 85.09 for RoBERTa-base.

All methods use the same data, backbone, batch size, and a fixed tuning budget of the same number of learning-rate and temperature trials each.

What counts as success. We beat the naive references if we match SimCSE's STS average within half a point at $C \le 0.7$, or exceed compute-matched SimCSE outright. We beat the field if we reach CompCSE's reported level at lower measured cost.

Ablations. Three ablations isolate the key ingredients:

  • $k=0$ tests input-level dropping.
  • $p=0$ tests forking without dropping.
  • Zeroing versus attention-hiding tests the form of the perturbation.

Why not just…?

Why not just fork late and use dropout alone? This is the real threat, because it is simpler and already saves compute. We expect it to fall short: sharing the lower-layer dropout sample should make the two views even more alike than SimCSE's, and DiffAug's analysis suggests that low view divergence is what limits dropout. The $p=0$ row of the sweep settles this directly. If dropout-only forking matches the full method, we will report that as the finding and drop the token-hiding claim.

Why not just delete words from the input, with rules protecting negations? Input deletion is the $k=0$ endpoint and saves compute the same way. The survey and self-training papers above show it flips meaning, and rule-based protection depends on brittle parsers, a weakness the file notes for SDA on informal text. The negation probe and the protected-token variant measure how much rules recover. If protected input deletion matches late forking, the depth argument is wrong.

Why not just apply EfficientCL-style cutoff in a single pass? Cutoff is cheap, but it zeroes values that later layers still read, and it is applied without regard to depth. EfficientCL's own CoLA drop suggests this erodes syntax. Our reimplementation of cutoff, plus the SentEval acceptability check described below, will show whether attention-hiding above layer $k$ avoids that cost.

Why not just train SimCSE for fewer steps? The compute-matched baseline answers this. If it matches us, the savings come from over-training in SimCSE, not from our method.

Proposed Experiments and Decisive Checks

We train on the 1M Wikipedia sentences and evaluate on the seven standard STS tasks (STS12 to STS16, STS-Benchmark, SICK-R) using Spearman correlation. We also report wall-clock time and peak GPU memory per epoch on a single GPU. To check that savings grow with model size, we repeat the key settings on BERT-large, where the multi-pass cost bites harder.

For the negation probe, we apply SDA-style single negation to held-out Wikipedia sentences and measure the fraction of sentences for which the encoder ranks the original closer to a paraphrase than to its own negation. We also track the probability that a training positive pair is "broken", meaning its branch outputs are farther apart than the anchor is from its negation. To address the grammatical sensitivity that EfficientCL lost, we fine-tune a linear probe on CoLA and compare against SimCSE.

Decisive checks. The claim is supported only if three things hold together:

  • Negation-probe accuracy rises steeply from $k=0$ and plateaus by middle depths.
  • STS peaks at an intermediate $k$ with $p>0$ and beats the $p=0$ row.
  • Measured compute at that peak is near the predicted $C(k,p)$ and below 0.7 of SimCSE.

If STS is best at $k=0$, or the $p=0$ row equals the full method, the core claim fails.