Teaching Intermediate Layers Which Edges Are Real: Occlusion-Aware Deep Supervision for RGB-Thermal Segmentation

A pane of glass is see-through to an ordinary camera but looks like a solid wall to a thermal camera, because long-wave infrared light (8–12 μm, the band that carries body heat) cannot pass through silicate glass. Systems that combine a color (RGB) camera with a thermal camera, called RGB-T systems, therefore receive contradictory evidence near glass. In the thermal image, the window frame or eyeglass rim shows up as a sharp, high-contrast edge that is not an object boundary at all. The real object behind the glass, such as a person in a lobby, is visible only as a faint, low-contrast outline in RGB. Current RGB-T segmentation networks learn to trust strong edges, so they draw boundaries in the wrong place, at the rim, and blur the boundaries that matter.

We propose a training-time fix that leaves the network architecture unchanged. Many segmentation networks already use deep supervision: extra losses attached to intermediate layers, so that each decoder stage is pushed toward the right answer. Normally those intermediate targets are just shrunken copies of the final label mask. We instead give the intermediate layers targets that say which modality can see what. Coarse stages predict a per-modality visibility map, meaning which camera is blocked in each region. Fine stages classify each boundary pixel into one of three types: a true object edge seen by both cameras, an occluder edge that appears only in thermal, or an object edge seen only in RGB. These labels are free if occlusions are synthesized with known geometry, using real thermal glass appearance cut from an existing RGB-T glass dataset.

Core Claim

Standard deep supervision tells every stage that every edge in the label mask is equally real and equally visible. Under spectral occlusion, meaning one camera is blocked by a material the other sees through, this is false. Our claim is that supervising intermediate layers with modality-specific visibility and edge-type targets removes false boundaries at occluder rims, and recovers low-contrast boundaries behind occluders, better than two cheaper fixes. The first is training on physically shaped occlusions alone. The second is adding a generic edge head. We further claim this costs nothing in accuracy on clean, unoccluded scenes. The new element is the supervisory target, not a fusion module. This differs from every cross-direction system in the file.

Spectral-Aware Global Fusion for RGB-Thermal Semantic Segmentation (SGFNet) splits RGB and thermal features into low and high frequencies with a discrete cosine transform and deeply supervises decoder stages. Its deep-supervision targets are downsampled semantic masks. Its design also favors modality-unique high frequencies, which is exactly what a cold glass rim is. We expect that preference to amplify occluder edges, and we test this directly.

RGB-T Semantic Segmentation With Location, Activation, and Sharpening (LASNet) attaches auxiliary location and edge heads to different depths. Its edge target, however, is a single undifferentiated boundary map, so a thermal-only rim and a true contour receive the same label.

Complementary Random Masking for RGB-Thermal Semantic Segmentation (CRM) simulates a missing modality by masking random 32-pixel patches, then enforces consistency between masked and clean views. It has no notion of material geometry or of the edges an occluder creates.

Glass Segmentation with RGB-Thermal Image Pairs exploits the glass opacity gap to segment the glass itself. It supervises only the output mask and never segments what lies behind the glass.

Research Program

Occlusion synthesis. The first component is a physically grounded occlusion generator. From the 5,551-pair RGB-T Glass Dataset released by Huo et al., we crop glass regions using their masks. In thermal, these crops show the near-uniform, partly reflective appearance of real glass. We paste them as contiguous panes, curved lens shapes, and frame-bounded windows into thermal images of MFNet and PST900. In the RGB image we add only a weak reflection overlay, because glass is transparent in visible light. Semantic labels are unchanged, since objects behind glass remain the correct answer. We also add the reverse case, smoke-like RGB attenuation with thermal intact, motivated by FlameFinder.

Because we place every occluder ourselves, we know its exact geometry. That knowledge gives us exact per-modality visibility masks and exact edge-type labels at no annotation cost. This replaces CRM's random squares with the contiguous, geometry-constrained occlusion zones that Background Modelling in Infrared and Visible Spectrum Video for People Tracking identified as the cause of tracking failures.

Scale-split supervision. The second component assigns targets by scale, following the observation that shallow-to-deep supervision naturally separates broad low-frequency structure from fine boundary detail (Intermediate Layer Supervision for Hierarchical Feature Learning). Occlusion regions are large and smooth, so visibility belongs at coarse stages. Occluder rims and low-contrast contours are thin, so edge typing belongs at fine stages. The training objective is

$$ L = L_{\text{seg}} + \sum_{s \in \mathcal{C}} \lambda_s\, L_{\text{vis}}^{(s)} + \sum_{s \in \mathcal{F}} \mu_s\, L_{\text{edge}}^{(s)}, $$

where the symbols are defined as follows. $L_{\text{seg}}$ is the usual final segmentation loss. $\mathcal{C}$ and $\mathcal{F}$ are the coarse and fine decoder stages. $L_{\text{vis}}^{(s)}$ is a two-channel binary loss predicting RGB-visible and thermal-visible regions at stage $s$. $L_{\text{edge}}^{(s)}$ is a three-class plus background cross-entropy over boundary pixels, distinguishing shared, thermal-only occluder, and RGB-only edges. $\lambda_s$ and $\mu_s$ are stage weights, which we fix to equal values to avoid adding another heuristic tuning knob.

The auxiliary heads are lightweight convolutions that are discarded at inference, so deployed cost equals the host network's.

Real-data annotation. The third component is a small real-data test set. We will annotate semantic masks of objects and people behind glass in a few hundred RGB-T Glass Dataset images. This "behind-glass" split addresses the inverse problem that Huo et al. leave open. It is also the only evaluation not built on our own synthesis, which matters because synthetic results could flatter the method.

Representation probe. Finally, we probe intermediate features with linear classifiers that predict edge type. This tests whether the supervision changes what the layers represent, rather than only what the output shows.

Baselines

The naive reference point has two parts. The first is the host network trained on RGB only. The second is the host network with four-channel early concatenation and no auxiliary losses. LASNet reports that simple concatenation and summation let noisy RGB corrupt thermal features, so we expect this to be the floor. Its false-edge rate at rims shows how bad the problem is without any intervention.

The strongest existing methods differ by benchmark. On MFNet the strongest reported number in the file is CRM at 61.2% mIoU, ahead of SGFNet at 60.1% mIoU (Spectral-Aware Global Fusion). On PST900 the strongest is the SAM2-based Segment Any RGB-Thermal Model with Language-aided Distillation at 89.88% mIoU, which adapts a large RGB-pretrained segmenter with low-rank adapters and language-guided distillation. On glass, the reference is Huo et al. at 93.80% IoU and 0.027 MAE on the RGB-T Glass Dataset. No reported result was found on Lacuna for any method on behind-glass semantic segmentation or on synthetic-occlusion versions of MFNet or PST900; we will produce those numbers ourselves.

For fairness, we graft our supervision onto the publicly reproducible host whose backbone matches CRM and SGFNet. We retrain those baselines under the identical schedule, crop size, augmentation, and GPU budget. Each method, including ours, receives the same small grid search over learning rate and auxiliary-weight scale. Results are averaged over three seeds.

Beating the strongest method means two things. The first is matching its clean mIoU within 0.5 points. The second is improving behind-glass mIoU and lowering the rim false-edge rate by more than the seed-to-seed spread.

Three ablations isolate the key ingredient. In the first, the occlusion-augmented data is used with standard deep supervision on downsampled masks; this is the key one. In the second, the same data is used with a single undifferentiated edge head, as in LASNet. In the third, only visibility supervision or only edge-type supervision is applied, and their stage placement is swapped, so coarse stages receive edge typing and fine stages receive visibility. This tests the scale-split argument.

Why not just…?

Why not just train on physically shaped occlusions with ordinary deep supervision? This is the most serious threat, and it may account for most of the gain. Exposure to realistic glass may be enough for the final loss to teach the network to ignore cold rims. The first ablation above settles it. If our method does not beat it on rim false-edge rate and behind-glass mIoU by more than seed noise, we will report the augmentation as the contribution and drop the supervision claim. Our reason to expect a gap is that the final mask loss gives no signal at a rim pixel when the rim falls inside an object's interior. Such a rim is simply "person" on both sides, so nothing tells intermediate layers that the strong thermal edge there is spurious. The edge-type target supplies exactly that missing signal.

Why not just add an edge-sharpening head, as LASNet or DB-MSMUNet do? Generic edge heads sharpen whatever edges appear in the label map. Examples are LASNet's edge sharpening module and the edge-enhancement path of DB-MSMUNet, which reached 89.47% Dice on low-contrast pancreas CT. Under occlusion, however, the most salient edges in the input are not in the label map, and the head has no class with which to reject them. Boundary-first designs like EBLNet, which subtracts edge features to supervise non-edge residuals and reached 90.28% mIoU on Trans10k, work in RGB alone and cannot tell that a glass interior lies at a different depth plane from the rim. The second ablation tests this directly by keeping everything else fixed.

Why not just fine-tune a foundation segmenter on RGB-T? The SAM2-based model is the strongest PST900 system in the file. Its authors found that adapters alone were insufficient across the RGB-thermal modality gap. They needed language-anchored distillation plus an auxiliary multi-scale head, which suggests that the intermediate-supervision route still matters at scale. Nothing prevents our targets from being applied to that model's auxiliary head, and if resources allow we will test exactly that. We expect its RGB pretraining to make it especially prone to the reverse failure, missing occluded thermal evidence, which the synthetic smoke test measures.

Proposed Experiments

The first check is clean-benchmark parity on MFNet and PST900, measured by mIoU against CRM, SGFNet, LASNet, and the SAM2 model. A regression of more than half a point would mean the auxiliary targets interfere with ordinary learning.

The second check uses synthetic-occlusion test sets built from held-out MFNet and PST900 images, with glass crops drawn only from a held-out portion of the RGB-T Glass Dataset to prevent leakage. On these sets we report three numbers: mIoU, boundary F-score within a thin band around true contours, and the rim false-edge rate. The rim false-edge rate is the fraction of predicted semantic boundary pixels lying near an occluder rim but far from any true contour. We predict that SGFNet will have a higher rim false-edge rate than LASNet despite its higher clean mIoU. That result would confirm the frequency-preference concern; finding the opposite would weaken the motivation.

The decisive check is the real behind-glass split. Gains there over the occlusion-augmentation-only ablation would show that synthetic edge-type supervision transfers to real glass, which the synthetic tests alone cannot show. We also report glass IoU and MAE on the standard RGB-T Glass test split against Huo et al., confirming that teaching the network to discount glass rims as semantic boundaries does not hurt its ability to find the glass itself.

Finally, linear probes on intermediate features should separate the three edge types well above chance only under our supervision. If that holds, the gain comes from changed representations rather than an output-level shortcut.