Text-embedding models turn a sentence into a vector so that sentences with the same meaning land close together. They break on scanned historical text. A misread character such as "thē" for "the" splits a word into rare fragments, and the corrupted sentence drifts away from its clean meaning. A cheap fix already exists: fine-tune the model contrastively, pulling each clean sentence toward a copy with randomly perturbed characters and pushing it away from the other sentences in the same training batch.
Separately, work on contrastive batch optimization shows that which sentences share a batch matters about as much as how many do. Batches deliberately filled with confusable "hard negatives" let small batches beat much larger random ones. The two lines meet at a specific failure. Under OCR noise, the sentences that look most confusable are often just misspellings of each other. A batch-mining method that trusts surface similarity will push apart texts that mean the same thing, and it will spend its batch on spelling variants instead of real semantic contrasts.
We propose to build contrastive batches by separating two things these methods currently merge. The first is how hard a pair is, which we measure in the model's own embedding space on the noisy text. The second is whether a pair is truly a negative, which we decide from clean-text meaning and source identity. We then test one claim: this noise-aware batch construction lets a single-GPU, small-batch run match or beat much larger random batches on OCR-robust retrieval, while naive hard-negative mining actively hurts.
Core Claim
Cheap Character Noise for OCR-Robust Multilingual Embeddings fine-tunes multilingual embedders (M-E5B, M-GTE) with in-batch negatives over clean and 5%-character-perturbed parallel sentences. It reports over 4 points gained on Blackletter retrieval, zero-shot transfer to historical Luxembourgish (HISTLUX), and no loss on clean similarity, all in about 20 minutes on one GPU. It uses random batches and does not study batch size or batch composition. That paper supplies our training recipe and evaluation, and it is the reference we must improve on.
Two papers supply the batch-construction method. Mini-Batch Optimization of Contrastive Loss proves that mini-batch contrastive training matches the full objective only if every possible batch is visited. It then curates high-loss batches, by loss ordering or spectral clustering, and raises CIFAR-100 retrieval from 70.29% to 74.71%. Breaking the Batch Barrier (B3) partitions a teacher-model nearest-neighbour graph into batches of mutually strong negatives. A batch size of 64 then outperforms models trained with batches 4 to 16 times larger on the multimodal benchmark MMEB.
Both assume that high similarity or high loss signals a useful negative. On OCR-corrupted text that assumption fails in a predictable way. A second noisy copy of the same source sentence, or a paraphrase with a different misreading, has high loss and high teacher similarity. Yet it is a false negative. We make this failure measurable and then remove it. Our claim is specific: with source identity and clean-text semantics used to veto false negatives, and noisy-space hardness used to rank true ones, OCR robustness at batch size 64 should equal or exceed random batching at 512, and B3 run unmodified on noisy text should underperform random batching.
Research Program
The training data follow Michail et al.: German and French parallel sentences, each clean sentence paired with one or more copies carrying 5% random character substitutions, insertions and deletions. Because the noise is synthetic, we know the clean source of every noisy string. That knowledge gives exact ground truth for which in-batch pairs are false negatives.
The first stage is diagnostic. For each batching strategy (random, loss-ordered, spectral clustering, B3 with a teacher run on the noisy strings), we log two quantities at each step. One is the fraction of in-batch negatives that are same-source or near-paraphrase pairs. The other is the share of the contrastive loss's repulsive gradient that flows through those pairs; the softmax weights make this share exactly computable. We expect hardness-seeking strategies to concentrate gradient on false negatives, and that concentration to rise with noise rate. This stage tests the core mechanism directly, before any benchmark number is involved.
The second stage is the method. Every $K$ steps we embed the training pool's noisy strings with the current student model and build a sparse k-nearest-neighbour graph. We remove every edge that meets either of two conditions. The first is that the two strings share a clean source. The second is that their clean versions exceed a similarity threshold $\tau$ under a frozen clean-text teacher, meaning the original M-GTE applied to clean text, where it is reliable. We then partition the pruned graph into batches with the same graph-partitioning step B3 uses. Each clean anchor is placed with its own noisy variants, which are treated as positives, never as negatives. The result is a batch whose negatives are hard because noise made them look alike, yet genuinely different in meaning. That is exactly the discrimination the benchmark CLSD demands. The added cost is one embedding pass over the pool every $K$ steps plus one partition, so it scales linearly with the pool size.
The third stage is the scaling study. We sweep batch sizes from 32 to 512 for each strategy, using gradient caching so that large batches fit on one GPU at identical per-step memory. We plot robustness against batch size. The question in the batch-optimization literature is whether curation substitutes for scale. Here we ask the same question under input corruption, where curation has a new way to fail.
Baselines
The naive reference point is the unadapted embedder: M-GTE and M-E5B with no fine-tuning, evaluated on every benchmark. It sets the floor and shows how much any adaptation matters.
The strongest existing method on this exact task is random in-batch contrastive fine-tuning with 5% character noise from Cheap Character Noise. The file reports its gain as over 4 points of Blackletter accuracy for M-GTE and no loss on clean similarity. No absolute CLSD or HISTLUX accuracy figures were found on Lacuna for this method, so we reproduce it ourselves under the same data, seeds, learning-rate grid and step budget as every other arm. We run it at every batch size in the sweep, including 512 with gradient caching. That makes "just use a bigger batch" a fully tuned competitor rather than a straw man.
The strongest existing batch-curation competitors are B3 and the spectral-clustering and loss-ordering selectors of Cho et al. We run each on the noisy training pool with the same total examples seen. B3 uses the frozen M-GTE as its teacher, applied to the strings it would actually see. Its reported results on other tasks, an MMEB average of 68.1 at 2B scale, and CIFAR-100 retrieval of 74.71% versus 70.29% for spectral clustering against vanilla training, are not directly comparable. They serve only as evidence that curation can beat scale in clean settings.
Two ablations isolate the key ingredient. One keeps our graph and partitioning but removes the false-negative veto. The other keeps the veto but draws batches at random from the vetted pool, which tests whether removing false negatives alone explains any gain. We count the method as beating the baselines if, at batch size 64, it exceeds the tuned random baseline at 512 on the average over CLSD's three noise types and on HISTLUX. The margin must exceed the seed-to-seed standard deviation across at least three seeds, and clean similarity must not fall.
Why not just…?
Why not just use a much larger random batch? This is the most serious threat. Larger random batches dilute false negatives, because a random draw rarely contains two copies of the same source, and they add more true negatives. If random batching at 512 matches our method at 64, the curation idea offers only a memory saving. That is still useful on one GPU, but it is a weaker claim, and we would report it as such. The batch-size sweep with gradient caching settles this question directly. The diagnostic stage also predicts where random batching should lose: on CLSD distractors that differ from the correct sentence by only a few words, and random batches almost never contain such pairs.
Why not just run B3 as-is? B3's teacher graph ranks candidates by similarity of the inputs it is given. On noisy strings, that similarity mixes meaning with spelling, and no step separates the two. Our first-stage diagnostic measures the false-negative share inside B3 batches, and the no-veto ablation is essentially B3 with a student-updated graph. If B3 unmodified already matches us, the veto is unnecessary. The experiment reports this either way.
Why not just drop same-source duplicates from each random batch? This one-line fix handles the most blatant false negatives, and it is plausibly enough when the noise is only synthetic character flips. It does not handle paraphrases, different sentences from the same article, or near-translations that clean-text meaning would flag. It also adds no hard negatives. The "veto plus random batches" ablation is exactly this fix, extended to meaning-level duplicates. If it matches the full method, the honest conclusion is that removing false negatives, not mining, drives the gain, and we would reframe the contribution around that.
Proposed Experiments
We evaluate on the benchmarks used by Michail et al. The first is CLSD, a cross-lingual semantic discrimination test in which the model must pick the correct translation over lexically similar distractors. It comes in three simulated noise conditions: minimal noise, Blackletter typeface and salt-and-pepper. The second is HISTLUX, real OCR from historical Luxembourgish newspapers, a language never seen during training. The third is clean semantic textual similarity, to catch any robustness tax. Metrics are discrimination or retrieval accuracy for CLSD and HISTLUX and Spearman correlation for similarity. We also report wall-clock time and peak memory, because the practical point is single-GPU training.
Four checks decide the outcome. First, the diagnostic must show that hardness-seeking batching raises false-negative gradient share relative to random batching; if it does not, the premise fails and we stop. Second, unmodified B3 should underperform tuned random batching on HISTLUX. This negative prediction distinguishes our account from "curation always helps." Third, our method at batch size 64 must match or beat random batching at 512 on the combined CLSD average and on HISTLUX. Fourth, the gain must survive the transfer from synthetic training noise to real OCR in HISTLUX. A method that wins only on synthetic CLSD would mean the veto is fitting the noise generator rather than OCR.
As a secondary sweep, we raise the training noise rate from 5% to 20%. We predict that the gap between noise-aware and naive mining widens as noise grows, since false negatives become more common. That trend would be the clearest evidence that the mechanism, not tuning luck, drives the result.