Steer, Don't Overwrite: Thermal Guidance for Frozen Open-Vocabulary Segmenters

A common way to make an image model work at night is to bolt a thermal camera onto it. Large pretrained vision-language models such as CLIP can label image regions with arbitrary words, but they were trained almost entirely on daylight RGB photos. When we add thermal input to such a frozen model, there are two broad options. We can mix thermal features into the model's internal features, which changes what it sees. Or we can let thermal input change only where the model looks, which leaves its features as they were. This proposal argues for the second option. It also argues for a narrow, gated channel for thermal content, opened only when the RGB view is actually failing. The expected payoff is a segmenter that keeps CLIP's open vocabulary and gets most of the night-time benefit of thermal sensing. Its central property is that, when the thermal sensor is broken, misaligned or fogged, it falls back exactly to the RGB-only model rather than below it.

The motivation comes from a tension in the literature that has not yet been tested in the open-vocabulary setting. OpenRSS adapts frozen SAM and CLIP to RGB-thermal input by inserting thermal "prompt" features into every layer of the frozen encoder. It reports large mIoU gains over RGB open-vocabulary methods. Separately, MultiCorrupt found in LiDAR-camera detection that early, tight cross-modal coupling is what breaks under sensor misalignment and dropout. In that benchmark, decoupled streams joined by sparse cross-attention (CMT) were the most robust. If that lesson carries over to RGB-thermal, OpenRSS-style layer-wise injection has two weaknesses. It should be fragile under thermal corruption. It should also push CLIP's features away from the region of feature space where they match text, which matters for classes never seen in training.

Core Claim

For a frozen open-vocabulary segmenter, the secondary modality's job splits into two parts. "Where" covers mask proposals and attention routing. "What" covers semantic content that is missing from a dark RGB image. These two parts have very different risk profiles.

We can inject "where" information as additive attention biases into the frozen CLIP encoder, following the mechanism of Side Adapter Network. That paper uses a small side network to predict masks and biases that make CLIP pool features only inside each mask. Bias injection leaves CLIP's value vectors untouched. Its worst failure is that CLIP looks at the wrong place, and it can be shut off completely by setting the bias to zero.

"What" information must actually add thermal content. We claim it should enter only once, late, through a gated cross-attention residual on each mask's final embedding. The gate is driven by a predicted per-region reliability score.

The testable consequence is a guaranteed floor plus a near-ceiling:

  • Floor: under thermal corruption, our model should never fall below RGB-only SAN.
  • Near-ceiling: under RGB darkness, it should recover most of OpenRSS's gain.
  • Vocabulary: its accuracy on unseen classes should match or exceed OpenRSS, because CLIP's features stay closer to their pretrained state.

Research Program

The model has three parts.

The first is a frozen CLIP image encoder that sees only RGB at low resolution, as in SAN, with its text encoder also frozen.

The second is a lightweight two-stream side network that takes high-resolution RGB and thermal in separate branches. It merges them through a cross-attention block in which RGB tokens act as queries and thermal tokens supply keys and values. This block is the standard cross-modal attention operation also used in the frame-event tracker of Zhang et al.. From this merged representation the side network predicts mask proposals and the per-layer attention biases fed into CLIP. It also predicts a reliability map $r \in [0,1]$ for each mask. That map is trained by perturbing the thermal stream during training and regressing whether the perturbation was applied. This is a simpler, supervised version of the per-teacher noise estimate in CUA-O3D.

The third is the late "what" channel. For mask $m$ with CLIP embedding $z_m$, we form

$$\hat z_m = z_m + r_m \, g_m \, W\,\mathrm{CA}(z_m, T),$$

where:

  • $T$ is the set of thermal tokens;
  • $\mathrm{CA}$ is cross-attention from the mask query to those tokens;
  • $W$ projects the result into CLIP's text-aligned embedding space;
  • $g_m$ is a learned scalar gate in the style of the gated cross-attention used for drug-target binding, where a gate blends cross-modal context with original features to prevent modality dominance.

Classification then compares $\hat z_m$ with text embeddings.

Two terms are added to the training loss. The first is the usual mask and classification loss. The second is a drift penalty that keeps $\hat z_m$ close to $z_m$ on well-lit images, so the residual is used only where it is needed.

Training uses masked-modality dropout and synthetic misalignment, which MultiCorrupt associates with CMT's robustness. All baselines that can use these augmentations get the same ones.

The analysis side of the project matters as much as the model. We will measure how much of thermal's benefit comes through masks and biases versus through the residual, separately for day, night and smoke conditions. We will also measure how far each fusion design moves CLIP's internal features away from their RGB-only values. This turns the observation that "frozen models inherit their pretraining ceiling" into a concrete quantity: feature drift versus unseen-class accuracy.

Baselines

RGB-only SAN. This is the do-nothing reference: SAN trained on the RGB half of the training data. No reported result on the RGB-thermal benchmarks was found on Lacuna. The file's reported SAN figure is 12.4 mIoU on ADE20K-847 (SAN), which we will reproduce as a sanity check of our implementation. It defines the floor that our model must never go below under thermal corruption.

OpenRSS. This is the strongest existing method on the question. It is the only open-vocabulary RGB-thermal segmenter in the file and is built on the same frozen foundation models. OpenRSS reports mIoU gains over RGB open-vocabulary methods of +12.1 on MF, +18.4 on MCubeS and +21.4 on Freiburg Thermal, training on KP with 1% of parameters trainable. No absolute mIoU for OpenRSS is in the file, so we will retrain it under the OpenRSS protocol rather than quote numbers.

Fair-comparison settings. OpenRSS and our model will be run with:

  • the same CLIP backbone;
  • the same KP training data;
  • matched trainable-parameter counts, around 1%;
  • the same augmentation suite, including modality dropout;
  • an equal hyperparameter-search budget.

What counts as beating OpenRSS. On clean data, matching it within one mIoU on unseen classes. Under thermal corruption, exceeding it at the highest severity, while staying at or above RGB-only SAN.

OpenRSS with a GCA-style gate. Here a gate is added to each thermal prompt. This baseline isolates whether location of injection matters or merely gating.

Late logit ensemble. This runs the RGB-only SAN alongside a thermal-only SAN, with per-pixel averaging.

FEANet. FEANet is closed-vocabulary and fully supervised. It reports +0.8 mIoU and +2.6 mAcc over prior work on MFNet. It serves as a reference for seen classes and for robustness of symmetric, ungated fusion, not as an open-vocabulary competitor.

Ablations of our model. Three ablations isolate the key ingredient: biases only, residual only, and both channels with the reliability term $r$ removed.

Why not just…?

Why not just train OpenRSS with modality dropout and misalignment augmentation? This is the most serious threat. MultiCorrupt credits masked-modal training as part of CMT's robustness, so augmentation alone might close the gap. Our view is that augmentation teaches the prompts to be small on average. It does not give a hard floor, and it does not stop layer-wise prompts from shifting CLIP's features on clean images. The experiment settles it directly: OpenRSS is given exactly our augmentation suite. If it matches us on corrupted-thermal mIoU and on feature drift, the bias-routing argument fails and we will report that.

Why not just fine-tune CLIP with low-rank adapters on RGB-thermal data? SAN was designed because fine-tuning CLIP for dense prediction degrades its zero-shot ability. Our drift-versus-unseen-accuracy measurement will show whether this happens here too. We include a LoRA variant in the drift analysis rather than as a headline competitor.

Why not just ensemble an RGB model and a thermal model at the output? An ensemble cannot share thermal's mask boundaries with CLIP's classification. It also needs a thermal model with an open vocabulary, which inherits all the distillation ceilings documented by OpenESS and CMEE. We expect the ensemble to be robust but weak at night. Our model should beat it there while matching its robustness floor.

Proposed Experiments

Protocol. Training follows OpenRSS: train on KP, evaluate zero-shot on MF, MCubeS and Freiburg Thermal. We report mIoU separately on seen and unseen classes and on day and night splits.

Robustness suite. We port MultiCorrupt's corruption taxonomy to the RGB-thermal setting at three severities:

  • RGB darkness;
  • missing thermal;
  • spatial misalignment (pixel shifts and small rotations between sensors);
  • temporal misalignment (frame offsets);
  • fog;
  • motion blur.

We score robustness with MultiCorrupt's Resistance Ability and Relative Resistance Ability metrics. The MultiCorrupt paper reports only rankings for these, not absolute values, so all numbers will be ours. This transfer of the protocol is itself a hypothesis. The LiDAR-camera finding may not hold for pixel-registered thermal, and a null result would be informative.

First decisive check. At the highest thermal-corruption severity, our model must stay within noise of RGB-only SAN. OpenRSS and the gated-OpenRSS variant must fall below it. If OpenRSS does not fall below RGB-only SAN, the motivating hypothesis is wrong.

Second decisive check. Under RGB darkness, the biases-only ablation should recover a minority of OpenRSS's gain, and the full model most of it. This would show that the late "what" channel is necessary and that routing alone is not enough when RGB carries little signal.

Third decisive check. On well-lit images, we measure the cosine distance between CLIP's final mask embeddings with and without thermal input. We expect this drift to be ordered: ours below OpenRSS, and OpenRSS below LoRA. Across all methods, we expect larger drift to go with lower unseen-class mIoU. That correlation would turn "pretrained ceilings" from a slogan into a measured trade-off.