Videos full of signs, labels, jerseys and storefronts contain words that answer the questions people ask about them. Current systems find this text with optical character recognition (OCR, software that reads characters from images) and then hand the model either a pile of per-word tokens or one summary sentence per sampled frame. Both choices are wrong for the same reason. A single sign seen across forty frames is one piece of information. It is blurred in some frames and sharp in others, it moves around the image, and it is often visible only between the few frames a video model samples. Our proposal is to collapse each physical text instance into one track token before fusing it with video. A track token records the best reading of the string over time, the instance's shape and motion, and the time interval during which it was visible. Video frames then attend to these tokens through an attention bias that depends on how far each frame is from each interval.
In two sentences: deduplicate scene text over time into per-instance tracks whose reading is a quality-weighted vote across frames, and let video frames attend to those tracks according to their visibility interval rather than forcing frame-by-frame pairing. This should fix three failures at once. It removes the token imbalance, because hundreds of repeated words become a few tracks. It recovers from motion blur, because the sharp frames out-vote the blurred ones. And it handles the timing mismatch between when text is visible and which frames the video encoder happens to sample.
Core Claim
The strongest existing system, the TCD network (Temporal Calibrating and Distilling for Scene-Text Aware Text-Video Retrieval), takes the opposite route.
- What TCD does. It packs OCR words into one templated caption per sampled frame ("There are scene texts: [words] in this frame"). This forces a one-to-one match with the 12 video frames. A cycle-consistency loss then repairs the resulting frame–caption mismatch, and learnable "clue tokens" filter OCR noise.
- What TCD reports. 60.4% R@1 on TextVR, against 19.2% for StarVR, along with large training-cost savings.
- Where TCD falls short. The fixed frame grid discards each instance's geometry and identity. It also double-counts text that persists across windows. Most importantly, it cannot represent text that is visible only between sampled frames, which a 12-frame grid will often miss in fast footage.
We claim the following. If OCR is run densely and grouped into instance tracks, and fusion uses an interval-aware attention bias instead of a fixed frame-to-caption pairing, the model should match or beat TCD with fewer text tokens. The gain should be largest on queries whose answer text is short-lived or blurred.
The idea combines three ingredients from different lines of work:
- Tracking. Following the morphological association of GLOMA, text is treated as a rigid object that can "jump" between frames.
- Down-weighting bad readings. We adapt the quality-aware temporal attention that SQUWA uses to suppress motion-corrupted segments of noisy physiological signals.
- Diagnosis. EgoTextVQA shows that frontier multimodal language models fail mainly at perceiving and timing scene text, not at reasoning about it.
Research Program
Building tracks. The first component converts dense OCR output into tracks. We run the same frozen OCR engine TCD uses, but on frames sampled densely (for example, several per second) rather than at the 12 sampled frames. Detections are linked into tracks by an association step. The default is GLOMA's Gaussian Wasserstein distance, which compares box position, scale and orientation and still works when overlap between consecutive boxes drops to zero. We combine it with string similarity, as the ICDAR 2021 video spotting winners did (ICDAR 2021 Competition on Scene Video Text Spotting).
Representing each track. Each track $k$ yields a single token with three parts.
- The string. Per-frame readings $r_{k,1..n}$ are embedded and combined with weights $w_{k,i} = \mathrm{softmax}_i(f(c_{k,i}, b_{k,i}))$. Here $c_{k,i}$ is the OCR confidence of reading $i$, $b_{k,i}$ is a sharpness score of the crop, and $f$ is a small learned network. This is temporal attention used as a learned vote.
- The geometry. A short summary of box shape, position and motion.
- The visibility interval. The start and end times $[s_k, e_k]$ during which the track was seen.
Fusing tracks with video. The second component fuses tracks with video frames. Sampled frame $t$ (at time $\tau_t$) queries the track tokens with an interval-aware bias:
$$a_{t,k} = \frac{q_t^\top k_k}{\sqrt{d}} - \lambda \, \mathrm{dist}\big(\tau_t, [s_k - \delta_-, \; e_k + \delta_+]\big).$$
Here $q_t$ is frame $t$'s query vector and $k_k$ is track $k$'s key vector, with dimension $d$. The function $\mathrm{dist}$ is zero inside the interval and grows linearly outside it. The learned margins $\delta_-, \delta_+$ absorb systematic lead or lag between when text is visible and which frames are sampled, and $\lambda$ sets how soft the window is.
A learned per-frame gate in $[0,1]$ mixes the fused text context back into the frame feature. This follows the gated cross-attention of Kim et al., which they introduced to stop an information-dense input from overwhelming the other. It addresses the negative transfer that StarVR/TextVR observed when all OCR tokens were fed in.
The bias does the job that TCD's cycle-consistency loss does, but structurally: text visible near a frame is preferred, and text far in time is softly excluded. We keep the clue-token distillation stage from TCD unchanged, so the comparison isolates the change in representation.
Using tracks with a language model. The third component is a serialized form for question answering. Each track is written as one line containing the string, its interval and a coarse location, then given to a multimodal language model under a fixed token budget. This tests whether the track abstraction helps even when a language model rather than a learned fusion layer consumes the text.
Baselines
Naive reference. A vision-only CLIP4Clip-style retriever with no scene text. Its absolute numbers are not given in the file, so we will re-run it. TCD reports that adding its Window-OCR Captioner to CLIP4Clip gave an average 13.4% boost. That gives the size of the effect that adding text should have, and our track tokens added to the same CLIP4Clip should produce at least that boost.
Strongest existing method. TCD itself, with 60.4% R@1 on TextVR as reported in its paper. We will reproduce it under identical conditions:
- the same OCR engine,
- the same CLIP backbone,
- the same 12 sampled frames,
- the same training epochs,
- a matched hyperparameter search budget.
The dense OCR pass costs extra preprocessing time, which we report separately. Beating TCD means a higher reproduced R@1 at equal or fewer text tokens per video. A tie at substantially fewer tokens also counts as a meaningful result. StarVR (19.2% R@1 as reported by TCD) is included as the unconstrained-concatenation reference.
Ablations. Two ablations isolate the key ingredients:
- Removing the interval bias (setting $\lambda = 0$). This tests whether timing-aware attention, rather than deduplication alone, carries the gain.
- Replacing learned quality voting with plain majority voting over strings within each track.
A third control gives TCD dense OCR packed into its per-window captions, which separates "more OCR frames" from "track structure."
Question-answering baselines. On EgoTextVQA, the reference is the frontier models' reported ~33–34% accuracy. The direct competitor is the same model fed raw per-frame OCR dumps at an equal token budget.
Why not just…?
Why not just give TCD dense OCR? This is the most direct threat, and the "TCD with dense OCR" control is designed to settle it. TCD's captioner is built around a fixed one-to-one frame-to-caption grid. Dense OCR either inflates the windows with repeated words or must be truncated to fit. If this control matches our model, then the benefit comes from coverage, not tracks, and we will report it that way.
Why not just feed timestamped OCR to a multimodal language model? EgoTextVQA shows that external OCR raises accuracy "significantly", so this is a real competitor for question answering. We do not argue against it. Instead, the comparison asks whether track serialization beats raw per-frame dumps at the same token budget. For retrieval over 10.5k long videos, running a language model per candidate video is impractical, so a learned fusion layer remains necessary there.
Why not just smooth OCR strings with majority voting? The voting ablation measures this directly. If majority voting matches the learned weighting, we adopt it as the simpler choice. We expect learned weighting to help most on the blurred, motion-heavy footage where the ICDAR 2021 results show end-to-end spotting dropping from 0.8502 detection F-score to 0.5308 end-to-end.
Why not just use higher-resolution crops? EgoTextVQA reports 20–30% gains from crops. This is complementary rather than competing: crops improve each single reading, while tracks fix duplication and timing across frames. We will run crops plus tracks against crops alone.
Proposed Experiments
Retrieval. The main experiment is on TextVR. We report R@1, R@5 and median rank, along with text tokens per video and training hours.
Diagnostic split. The decisive check is a split built from dense OCR of the TextVR videos. It selects queries whose matched text track is visible only in frames that the 12-frame sampler skips, or only in low-sharpness frames. Our method should show its largest margin over TCD on this split and roughly parity elsewhere. If the margin is uniform, the interval mechanism is not doing what we claim.
Controlled shift. We shift the frame sampler's timing relative to the track intervals. The learned margins $\delta_\pm$ should absorb the shift, while a model with $\lambda = 0$ should degrade.
Question answering. On EgoTextVQA under its real-time causal protocol, we compare serialized tracks against raw OCR dumps. We also report M4-ViteVQA, for which no reported result was found on Lacuna, so all numbers there will be our own reproductions.
Tracker quality. We measure how sensitive downstream gains are to tracking quality, using ICDAR2015 Video, where GLOMA reports 56.0 MOTA. We swap in a weaker IoU-based tracker and check whether retrieval gains shrink in proportion. A large shrink would show that the benefit depends on correct instance grouping.
References
- Temporal Calibrating and Distilling for Scene-Text Aware Text-Video Retrieval
- GLOMA: Global Video Text Spotting with Morphological Association
- SQUWA: Signal Quality Aware DNN Architecture for Enhanced Accuracy in Atrial Fibrillation Detection from Noisy PPG Signals
- EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering
- ICDAR 2021 Competition on Scene Video Text Spotting
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- A large cross-modal video retrieval dataset with reading comprehension