Word-Level Late Interaction for Retrieving OCR-Corrupted Documents

Scanned archives, clinical records and historical newspapers are searched through text produced by optical character recognition (OCR), the software that reads characters off page images. That text is full of misread characters, split words and merged words. Two problems then arrive together. Users type modern, clean queries that share few words with the archaic document text, which is the vocabulary mismatch problem. And the document text itself is corrupted in ways that break the models built to solve that mismatch. Retrieval models that compare meaning rather than exact words start by cutting text into subword pieces. A misread word like "frncture" gets cut into several rare fragments that no longer resemble "fracture" to the model. OHRBench finds that moving from mild to severe character-level OCR noise cuts most dense retrievers' performance by nearly half. On degraded government scans, DocDeg finds that the text route only beats an image-based retriever when it uses a 90-billion-parameter model to do the OCR.

We propose to fix this at the level of the matching unit. A late-interaction retriever keeps one vector per token and scores a document by letting each query token find its best-matching document token. We keep that design, but on the document side we regroup the subword vectors into vectors for predicted clean word units. The regrouping comes from a small boundary predictor trained on synthetically corrupted text, where the true word boundaries are known. We then adapt only the document encoder to character noise, with an anchor that keeps clean documents where they were. The bet is that most of the OCR damage to meaning-based retrieval comes from fragmentation: a corrupted word becomes several unrelated vectors. If each corrupted word is reassembled into one contextual vector before matching, a cheap OCR engine plus a small retriever should approach the accuracy of the 90B pipeline while still bridging synonyms.

Core Claim

Current fixes each address one layer of the problem. Cheap Character Noise for OCR-Robust Multilingual Embeddings fine-tunes single-vector encoders on text with 5% random character edits. It gains over 4 points on Blackletter retrieval with no cost on clean text. However, it compresses each passage into one averaged vector. That averaging is the "bag-of-embeddings" loss of word order and local context documented by Fu et al., and it is tested only on short sentences. CAPOT makes dense retrieval robust to typos, but only on the query side, with a frozen clean document index. OCR archives have the opposite asymmetry. Token-level late interaction as in ColBERTv2 and its fast engine PLAID avoids single-vector compression, but it has never been examined on OCR text. Its scoring is also an unordered "bag of token matches," so OCR fragments such as "nit" or "cond" can create spurious high-scoring matches.

Our claim is testable. Under vanilla late interaction, the loss in retrieval score should grow with the number of subword pieces a corrupted document word shatters into. Pooling to predicted word units should flatten that curve, and noise training without pooling should not. If the curve is already flat for vanilla ColBERT, or if noise training alone flattens it, the pooling idea is wrong and we will report that.

Research Program

The system has three parts. The first is a character-level boundary predictor, a small model that reads OCR text and tags where clean words begin. This is needed because OCR also corrupts spacing: "frac ture" splits one word and "ofthe" merges two, so whitespace is an unreliable guide. We train it on clean text corrupted with the cheap random insertions, deletions and substitutions of Michail et al., plus random space insertion and deletion. Because we generate the noise, the alignment back to clean word boundaries is known exactly. This is the piece that the sub-document span methods SenClu and C-Top2Vec lack. Both show that moving from word counts to contextual spans helps; SenClu raises 20Newsgroups PMI coherence from 0.35 for LDA to 0.79. But both assume sentence punctuation and token boundaries that OCR destroys. Here they act as a constraint on our design, not as a method we reuse.

The second part is the scoring. Let $q_i$ be the contextual vector of query token $i$. Let $u_j$ be the pooled document vector for predicted word unit $j$, computed as a learned attention-weighted average of the subword vectors inside that unit. The score is

$$s(q,d) = \sum_i \max_j \; q_i^\top u_j .$$

This is ColBERT's rule applied to reassembled words instead of raw fragments. A side effect is that the document index shrinks, because it stores one vector per word rather than one per subword. That matters when OCR text has inflated subword counts.

The third part is document-side adaptation. We freeze the query encoder and fine-tune the document encoder. The contrastive loss pulls the representation of a noised document toward its clean query. An anchoring term, borrowed from CAPOT but applied in the reverse direction, keeps clean-document unit vectors close to those of the original frozen model:

$$\mathcal{L} = \mathcal{L}_{\text{contrast}}(q, \tilde d) + \lambda \sum_j \lVert u_j(d) - u_j^{0}(d) \rVert^2 .$$

Here $\tilde d$ is a noised document, $u_j^{0}$ is the frozen model's unit vector, and $\lambda$ sets the strength of the anchor. Synthetic training data comes from MS MARCO-style query-passage pairs with injected character noise. Following Michail et al.'s finding that generic random noise transfers better than realistic OCR simulators, we use random edits and do not model specific OCR engines.

The vocabulary-mismatch side comes from the pretrained contextual encoder, which already matches synonyms on clean text. Yang et al. show that many correct answers share no content words with the query. That is exactly why fuzzy string matching alone cannot be the answer.

Baselines

The naive reference point is BM25 over the raw OCR text. On BL19, the British Library's 18th- and 19th-century collection, Datta et al. report BM25 at MAP 0.4993, recall 0.5942 and nDCG 0.5653. We will rerun it on our exact document versions rather than reuse these figures.

The strongest vocabulary-side baseline on BL19 is their Fiction_RLM query expansion. It uses relevance-based language models, which expand the query with single words drawn from pseudo-relevant documents, here trained on 19th-century fiction. It reports MAP 0.5743 and recall 0.6944. It is the strongest competitor on the mismatch question. Beating it means higher MAP on the same 35 queries and judgments, with the gap growing as injected noise increases.

The strongest OCR-side baseline is the DocDeg pipeline of Llama 3.2 90B for OCR followed by Qwen2 embeddings, which reports MRR 0.4852 over all four degradation levels. ColPali, which retrieves directly from page images, reports 0.2471. We cannot rerun a 90B OCR pass cheaply, so we will compare against their reported number on the identical DocDeg splits. Our system will run a small OCR engine plus a retriever of about 110M parameters. The claim counts as successful if we reach within a few MRR points of 0.4852 at a small fraction of its compute and its reported 9.5 GB per 1,000 documents of storage.

For OHRBench, no reported absolute retrieval number was found on Lacuna, only relative drops. We will therefore run all baselines ourselves on its semantic-noise tiers.

The fair-comparison baselines share our backbone, training data, noise generator and GPU budget. The first is a single-vector encoder trained exactly as in Michail et al. The second is vanilla ColBERTv2 zero-shot. The third is ColBERTv2 fine-tuned with the same noise and anchor but without unit pooling; this is the key ablation that isolates our contribution. The fourth is ColBERTv2 with whitespace pooling instead of learned boundaries, which isolates the boundary predictor. All receive the same hyperparameter search budget.

Why not just…?

Why not just fine-tune a single-vector encoder with cheap character noise, as Michail et al. did? This is the real threat. It is cheap, it works on sentence-level tasks, and it costs nothing on clean text. Their evidence stops at short sentences, however. For long documents where only a few words match the query, averaging dilutes those few matches, as Fu et al.'s pooling analysis predicts. We run this exact baseline with the same noise and budget. If it matches us on DocDeg levels 2–3 and on noised BL19, the multi-vector machinery is not justified.

Why not just correct the OCR text first and then use a standard retriever? Post-OCR correctors such as Levenshtein OCR and glyph-embedding correction are natural fixes. But OHRBench shows that even the best OCR outputs leave at least a 14% F1 gap to clean text in RAG. Correctors also tend to "fix" historical spellings and domain terms into the wrong modern word. We include correction followed by ColBERTv2 as a baseline. Our method and correction can also be stacked, and we will report that combination too.

Why not just run BM25 with fuzzy or character n-gram matching plus query expansion? Character n-grams handle "pariament" versus "parliament" but not zero-overlap paraphrase. Unigram expansion drifts, and Datta et al. report that it hurts precision on factual queries. Since this is cheap, we build a hybrid of character-n-gram BM25 with Fiction_RLM expansion. If it matches us on BL19, where the mismatch is mostly cultural vocabulary rather than paraphrase, we will say so and restrict our claim to paraphrase-heavy settings.

Why not just noise-train ColBERT and skip pooling? This may already work, because contextual vectors of fragments carry information about their neighbors. The no-pooling ablation above settles the question directly.

Proposed Experiments

We evaluate on DocDeg by degradation level, OHRBench retrieval by semantic-noise tier, BL19 with controlled character-noise injection at 0%, 5%, 10% and 20%, and HISTLUX for zero-shot transfer to real historical Luxembourgish OCR. The metrics are MRR, MAP, recall and nDCG, plus index size and query latency.

The decisive check is the fragmentation curve. We align noised documents to their clean originals. For each query term, we record how many subword pieces its corrupted document counterpart became. We then plot how much that term contributes to the relevant document's score, relative to the clean version of the same document. The hypothesis predicts a falling curve for vanilla and noise-trained ColBERT and a flat curve for unit pooling.

The second check is the clean-text tax. Scores on DocDeg level 0 and on uncorrupted BL19 must not fall below those of the unadapted model.

The third check measures spurious fragment matches. We count top-ranked non-relevant documents whose score is dominated by matches to single short fragments. The count should fall with pooling.

The fourth check is boundary sensitivity. We degrade the boundary predictor on purpose and measure how quickly retrieval gains vanish. This tells us whether the learned segmenter is doing the work or whether whitespace alone suffices.