Reading Only What the Heat Shows: Honest Text in Thermal-to-Visible Video Translation

Thermal cameras see in the dark because they record heat that objects give off, not light bouncing off them. Systems that turn thermal video into normal-looking daytime color video let ordinary vision models work at night. There is one category of content these translators cannot honestly recover: printed text. A license plate's characters and the plate behind them are usually the same temperature, so the letters often leave no trace in the thermal image. The obvious move is to bolt a text-recognition loss onto the translator so the output text becomes legible. That move is dangerous. When the input contains no evidence of the characters, a translator rewarded for legibility will invent plausible but false plates and signs, and it will do so convincingly.

We propose to use a frozen text recognizer as two things at once. First, it checks whether each piece of output text is grounded in the input. Where the thermal frame carries character-level evidence, such as heated LED signs, embossed plates, or paint with a different emissivity, the output reading must match it. Where it carries none, the output must stay unreadable rather than guessing. Second, the recognizer acts as a temporal referee: the reading of a tracked text region should not change from frame to frame. Neither signal needs paired thermal–visible video, which does not exist at scale. The temporal signal comes from the unpaired thermal video itself. The evidence signal comes from reading the thermal input directly.

Core Claim

Adding an OCR perceptual loss to cross-modal video translation, as is done for RGB text generation, will raise legibility. In regions with no thermal evidence it will also raise the rate of confidently hallucinated text. An evidence-gated, temporally consistent OCR objective can improve faithful reading where evidence exists while keeping hallucination at or below the level of a translator with no text supervision. The gap in the literature is real. ROMA translates unpaired nighttime infrared video to daytime visible video. It matches global, local, and temporal similarity between features from a vision transformer (an image model that processes the frame as a grid of patches), and it has no character-level term. OCR-guided losses in AnyText, OCR-VQGAN, LPDGAN, and learned compression with text enhancement all work on single RGB images. In each of those settings a ground-truth or source rendering of the text exists, so legibility and correctness coincide. In thermal-to-visible translation they come apart. That separation is what this proposal addresses.

Research Program

The generator $G$ is a ROMA-style unpaired translator mapping thermal frames $x_t$ to visible frames $y_t$. We reuse its objectives and add text terms on top. A text detector run on the outputs proposes text regions. Optical flow computed on the thermal sequence links these regions across frames, which works because motion is visible in thermal even when the characters are not. Each region is rectified and passed through a frozen recognizer $R$, a CRNN-style model as in LPDGAN, which returns per-position character probabilities $R(\cdot)$.

The evidence score $e \in [0,1]$ for a region comes from running $R$ on the contrast-normalized thermal crop under both white-hot and black-hot polarity. This handles the polarity inversion described in the thermal literature. The score is the larger of the two mean per-character confidences, calibrated on a small annotated set. The added loss is

$$ \mathcal{L}_{text} = \lambda_{tmp}\,\mathrm{KL}\big(R(y_t^{r}) \,\|\, R(y_{t+k}^{r})\big) + \lambda_{ev}\Big[e\,\mathrm{CE}\big(R(y_t^{r}), \hat{s}(x_t^{r})\big) + (1-e)\,\max\big(0,\, H_0 - H(R(y_t^{r}))\big)\Big]. $$

Here $y_t^{r}$ and $x_t^{r}$ are the output and thermal crops of tracked region $r$, $k$ is a frame offset, and $\hat{s}$ is the thermal reading. CE is cross-entropy, $H$ is the per-position entropy of the recognizer's output, and $H_0$ is a floor below which an ungrounded region counts as "confidently read."

The first term is the temporal OCR consistency that the literature lacks. Without the second term it is risky, because a translator can satisfy it by hallucinating the same false string in every frame. The bracketed term therefore splits the regions. Grounded regions must read like their thermal evidence. Ungrounded regions are penalized only for being confidently readable. They are not forced toward any rendering, so the existing ROMA losses still decide how the plate looks. The term operates on recognizer logits rather than decoded strings because logits give smooth gradients, the reason Lai et al. supervise at that level.

Video-editing data scarcity sets the design constraint. Like the frozen-teacher approaches surveyed for video editing, including the factorized distillation of Singer et al. (link), every supervisory signal here comes from frozen experts and from the structure of unpaired video, never from paired outputs.

For evaluation we build ThermalText, a small set of annotations on the hardware-aligned color–thermal sequences of KAIST and on FLIR ADAS imagery. Annotators mark text regions and transcriptions in the RGB view. They also label from the thermal view alone whether each region is thermally evident. This yields ground truth that no existing benchmark provides: which text should be recoverable and which should not.

Baselines

The naive reference is the raw thermal video read directly by the recognizer, together with ROMA with no text term. ROMA is the strongest existing translator. On InfraredCity-Lite it reports 50.1% YOLOv3 AP on translated video, against 25.0% for raw nighttime infrared and 32.2% for I2V-GAN (ROMA). There is no reported result on Lacuna for text recognition or hallucination on any thermal-to-visible translation output, so ROMA's text behavior must be measured here. We retrain ROMA from its released recipe on the same InfraredCity and KAIST training frames and with the same compute budget as every variant.

The strongest competitor on the text question is the obvious idea: ROMA plus an ungated OCR feature-and-logit perceptual loss in the style of AnyText and LPDGAN, which pushes output text toward confident readability. LPDGAN shows how much such losses help when the text is truly present: on LPBlur it reports a Text Levenshtein Distance of 0.81 against 1.13 for the next model. That figure is RGB-only and is not comparable to our setting, but it shows this baseline is strong at legibility. We give it the same hyperparameter search budget as our method.

The ablations isolate the two ingredients: the temporal term alone, the evidence gate alone, and the gate with the thermal reading replaced by a constant $e$. Beating the field means two things on ThermalText. Character error rate on evident regions must be at least as good as the ungated OCR loss. The hallucination rate on non-evident regions must be no higher than plain ROMA and clearly below the ungated loss. YOLOv3 AP on InfraredCity-Lite must stay within a point of ROMA's.

Why not just…?

Why not just add an OCR perceptual loss to ROMA? This is the main hypothesis under test, not a strawman. Ungated legibility rewards are well supported on RGB, as the AnyText sentence-accuracy gains of 72.39% versus 59.21% for TextDiffuser show. The physics of emittance means the target text is often absent from the thermal input. We predict this baseline wins on evident regions and loses badly on hallucination. If its hallucination rate turns out no worse than ROMA's, our gate is unnecessary and we will report that.

Why not just detect text in the output and blur it? This is a real threat on safety grounds. Suppressing every text region guarantees zero hallucination at almost no cost. It also throws away text that is genuinely recoverable from heat. ThermalText's evident-region split measures exactly how much is lost. If evident regions turn out to be very rare in KAIST and FLIR, blur-and-abstain is the right answer, and the contribution shrinks to the evidence estimator, which decides what to blur.

Why not just vote over per-frame OCR readings afterwards? Majority voting across frames removes reading flicker for a downstream reader. A cascaded pipeline cannot restore strokes the generator already destroyed, as LPDGAN's baseline gap illustrates. Voting over hallucinated readings also returns a stable but wrong string with more confidence. We run this post-hoc voting on ROMA outputs as an extra baseline. It should fix the stability metric but not hallucination or faithful-read error.

Why not just fine-tune a large video editor such as EditVerse on thermal video? Paired night-thermal/day-visible video cannot be captured. EditVerse is trained only on RGB and needs about 118 seconds and 30GB of memory per 360p edit. RGB foundation models also degrade on infrared statistics without adaptation (IRSAM). None of this addresses whether the text it produces is grounded, so we do not treat it as a primary baseline.

Proposed Experiments and Decisive Checks

We train on InfraredCity and the KAIST thermal sequences, then evaluate on ThermalText and InfraredCity-Lite. ThermalText metrics are character error rate against RGB transcriptions on evident regions and hallucination rate on non-evident regions. A region counts as hallucinated when the recognizer reads three or more characters above a calibrated confidence. A third metric is temporal reading stability, the fraction of consecutive frames in which a tracked region's decoded string changes. Fidelity uses FID and YOLOv3 AP on InfraredCity-Lite. We also run a human study in which raters, shown the thermal input, judge whether each rendered string is supported by it.

The decisive check is a two-axis plot of faithful-read error against hallucination rate. The ungated OCR loss should move down on error and up on hallucination. Our method should move down on error with hallucination flat or lower. A second check stratifies results by thermal contrast inside the region, to confirm the gate tracks the physics and does not simply suppress all text. The proposal fails if the evidence score is no better than chance at predicting human evident labels, or if the temporal term alone matches the full method on hallucination.