When a camera sweeps past a shop sign, no single frame may show the whole word clearly. The first letters are sharp early, the last letters come into focus later, and a passing pedestrian hides the middle for a few frames. Current video text systems either read every frame independently or pick the one "best" frame and discard the rest. We propose a third option. Once a text instance has been tracked, the recognizer's internal character evidence is passed forward and backward along the track, in the spirit of bidirectional video restoration networks. Every frame's reading can then draw on sharper views that come both before and after it. We apply this to recognition features, not to pixels.
The claim is only worth making if the measurements are honest. Published video text scores mix detection, tracking, synthetic pre-training and recognition into one number. We therefore pair the method with a locked evaluation protocol. The protocol holds tracks, crop extraction and string-matching rules fixed, so that recognition gains can be attributed to the fusion method. It also reports separately how much of each method's accuracy survives when it is trained only on synthetic video.
Core Claim
The ICDAR 2021 video text competition shows the size of the problem. The winning system scores a per-frame detection F-score of 0.8502 but only 0.5308 on end-to-end sequence spotting on the Large-Scale Video Text Dataset (LSVTD) (ICDAR 2021 Competition on Scene Video Text Spotting). Much of that loss is reading blurred, partly occluded text over time.
Our claim is that recognition over a track benefits from temporal order, not just from pooling many frames. Adjacent frames are geometrically similar. Evidence can therefore be aligned in small steps and chained across the track, whereas a direct comparison of distant frames must bridge large viewpoint changes. Running this chain in both directions removes the "warm-up" weakness of causal models, in which the first frames of a track have no history to draw on.
This claim is falsifiable against a very simple competitor: averaging per-frame character probabilities, or a learned attention pool that ignores order. If either matches bidirectional propagation, the order-aware part of the idea is not needed, and the protocol will show it.
Research Program
The method borrows its structure from BasicVSR. BasicVSR super-resolves video with two recurrent streams, one running forward and one backward, and aligns features between neighboring frames using optical flow. It outperforms EDVR by 0.33 dB PSNR on REDS4 (a standard video super-resolution test set), and its feature-level alignment alone is worth over 1 dB. BasicVSR is optimized for pixel fidelity on natural scenes and never measures whether text becomes readable.
TSRN shows why that matters. It super-resolves still text crops for the recognizer rather than for PSNR. It uses bidirectional LSTMs over the horizontal character sequence, and that component was its most important, adding 4.9% average recognition accuracy. We keep TSRN's lesson that the objective should be readability, and move the bidirectional pass from the spatial axis to the time axis.
The pipeline has four steps. First, each tracked crop is rectified to a canonical horizontal strip using the track polygon, with a thin-plate-spline warp as in TSRN. Second, a shared convolutional recognizer encoder maps each strip to a sequence of column features $e_t$ for frame $t$. Third, a small learned module estimates a one-dimensional horizontal offset between the column features of adjacent frames and warps the previous state to match. This offset absorbs residual misalignment, playing the role that optical flow plays in BasicVSR. Fourth, two recurrent streams are run:
$$h^{f}_t = F\big(\mathcal{W}(h^{f}_{t-1}),\, e_t\big), \qquad h^{b}_t = B\big(\mathcal{W}(h^{b}_{t+1}),\, e_t\big),$$
Here $h^f_t$ and $h^b_t$ are the forward and backward hidden states at frame $t$, $\mathcal{W}$ is the learned column warp, and $F$ and $B$ are residual convolutional blocks.
A connectionist temporal classification (CTC) head reads the concatenation $[h^f_t, h^b_t]$. CTC is a loss that lets a recognizer output a character string without per-character position labels. The head produces a transcript for every frame, which is what frame-level spotting metrics score. A track-level transcript is taken from the frame with the lowest CTC loss after fusion.
We add one mechanism from IconVSR. When the learned warp's confidence drops, which usually signals an occlusion, the stream is refilled with features from the highest-confidence frame nearby.
The evaluation protocol follows lessons from other fields. NerfBaselines found that the choice of image downscaling method and bit depth alone can reorder leaderboards. We therefore fix crop resampling (bilinear, 32×128 strips) and color depth for every method. BioEL showed that strict versus relaxed string matching changes conclusions. We therefore fix and report both case-insensitive alphanumeric matching and exact matching. Following Open Graph Benchmark, splits are immutable and released with loaders.
The protocol has two tracks. In the oracle-track condition, ground-truth track boxes are supplied, which isolates recognition. In the end-to-end condition, all methods share one fixed tracker, so recognition gains can be read off without confounds from tracking.
The sim-to-real question is built into the protocol. Every method is trained three ways: on synthetic tracks generated with FlowText, on real training tracks, and on synthetic tracks followed by real fine-tuning. FlowText renders text onto ActivityNet backgrounds and propagates it with flow-fitted homographies and directional motion blur. We report the fraction of the real-trained accuracy that the synthetic-only model retains.
Baselines
The naive reference point is per-frame reading. The same recognizer encoder and CTC head read each rectified crop independently, with no temporal information. Every gain is measured against this.
The strongest selection competitor is winner-frame reading in the style of YORO. YORO scores frame quality with a teacher-student network and recognizes only the best frame of each stream. Its recognition stage is about 71 times faster than reading every frame, and it beats voting when most frames are poor. We reimplement its quality scorer on our encoder so that selection is the only difference. We found no reported numerical F-score for YORO on Lacuna.
The strongest order-free competitors are posterior averaging and attention pooling. Posterior averaging aligns per-frame CTC probability columns and averages them. Attention pooling is a transformer over the track's frames without temporal position encodings, followed by the same head. These competitors directly test whether order matters.
The strongest pixel-restoration competitors are BasicVSR retrained on text crops with the recognition loss, followed by the recognizer, and TSRN applied frame by frame.
The strongest full systems are TransDETR and GLOMA. TransDETR passes text queries through time in a transformer and reports 72.8% IDF1 for spotting and 47.7% MOTA for tracking on ICDAR2015 Video. GLOMA matches detections against a causal memory of past frames using a Gaussian Wasserstein distance and reports 56.0 MOTA on ICDAR2015 Video. These enter only the end-to-end condition, with our recognizer placed on top of their tracks.
The key ablations are the forward-only stream (which isolates bidirectionality), removing the learned warp (which isolates alignment), and removing refill.
All learned methods share the encoder, the training data for each regime, a GPU-hour budget, and an equal random-search budget over their own hyperparameters. Tuning is done on a validation split carved from the training videos, never on test.
Beating the field means two things. First, a word-accuracy gain over attention pooling that exceeds seed-to-seed variation on both ICDAR2015 Video and LSVTD in the oracle-track condition. Second, an IDF1 above TransDETR's 72.8% on ICDAR2015 Video when paired with a shared tracker.
Why not just…?
Why not just pick the sharpest frame, as YORO does? Selection only works when some frame shows the entire word clearly. It fails by construction when different characters are sharpest in different frames, or when occlusion never fully clears. It is also much cheaper, and it may be sufficient on short, easy tracks. We stratify tracks by the peak per-frame recognizer confidence. If winner-frame reading ties with fusion on high-peak tracks and loses only on low-peak ones, that result is useful. It would support a hybrid that runs fusion only when no good frame exists.
Why not just average character probabilities, or pool frames with attention? This is the real threat. If posterior averaging or order-free attention matches the bidirectional model, the temporal-order claim fails and we will say so. Our reason to expect a gap is BasicVSR's evidence that alignment chained through neighboring frames gives over 1 dB. A set-pooling model must instead align each pair of frames directly. We test this by binning tracks by viewpoint change between their first and last frames, measured from track-polygon homographies. The gap should grow with viewpoint change if our reasoning is right, and vanish otherwise.
Why not just super-resolve the video with BasicVSR and then read it? Pixel restoration optimizes the wrong target. TSRN's results show that readability-driven design beats generic super-resolution by more than 5% on the ASTER recognizer. Restoring full frames is also expensive, and restoration trained on synthetic blur is exactly what TextZoom found to transfer poorly to real optics. The recognition-loss BasicVSR baseline and the retained-accuracy ratio across training regimes settle this question directly.
Why not just fine-tune TransDETR with heavier augmentation? TransDETR already propagates queries through time, but mostly forward. Its recognition is entangled with its detection. We include it in the end-to-end condition with and without our fusion head on its tracks. Under this design, the gain from augmentation and the gain from backward context can be read off separately.
Proposed Experiments
The first decisive check is the oracle-track comparison on ICDAR2015 Video, ICDAR2013 Video, and LSVTD, including its non-Latin clips. It reports word accuracy and character error rate under both matching rules, over five seeds. The critical contrasts are bidirectional against attention pooling, and bidirectional against forward-only. The forward-only contrast is reported separately for the first quarter of each track, where the warm-up effect described in Revitalizing Legacy Video Content should concentrate.
The second check is the sim-to-real audit. We compute the retained-accuracy ratio for every method trained on FlowText-only tracks. We also run a mini-experiment inspired by the planar-surface limitation of FlowText: tracks on curved or flexible surfaces are hand-tagged in the test sets, to see whether any method's synthetic-to-real loss concentrates there.
The third check is end-to-end. Our fusion head is placed on GLOMA-style and TransDETR tracks, and we report IDF1 and MOTA on ICDAR2015 Video against TransDETR's 72.8% IDF1 and GLOMA's 56.0 MOTA. On LSVTD we report sequence-level spotting F-score against the competition's 0.5308, noting that the winner used unreleased data, so that comparison is only indicative.
Runtime per frame is reported for every method next to its accuracy. That way, a gain that costs more than YORO's selection speedup is visible as such.
References
- ICDAR 2021 Competition on Scene Video Text Spotting
- BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond
- Scene Text Image Super-Resolution in the Wild
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods
- BioEL: A Comprehensive Python Package for Biomedical Entity Linking
- Open Graph Benchmark: Datasets for Machine Learning on Graphs
- FlowText: Synthesizing Realistic Scene Text Video with Optical Flow Estimation
- Efficient Video Scene Text Spotting: Unifying Detection, Tracking, and Recognition
- End-to-End Video Text Spotting with Transformer
- GLOMA: Global Video Text Spotting with Morphological Association
- Revitalizing Legacy Video Content: Deinterlacing with Bidirectional Information Propagation