Two Probes for Where Multimodal Pseudo-Labels Go Wrong

Self-training is the standard way to use unlabeled data: a model labels the unlabeled images itself and then trains on those guesses, called pseudo-labels. When the model is a vision transformer that fuses a color camera with a second sensor such as a thermal camera, its pseudo-labels tend to fail for one of two reasons. The first is that one sensor is blind, as RGB is at night, and its errors are carried into the fused prediction. The second is that the transformer cuts the image into coarse patches and averages information across them, so object boundaries come out blurred. Both sensors are fused inside the same backbone, so this blur is shared by both of them. Current filters do not tell these two failures apart. Confidence thresholds miss confidently wrong boundary pixels. Voting between modalities catches blind-sensor errors, but it cannot catch blur that both streams inherit from the same tokenizer. Generic uncertainty estimates discard boundary pixels wholesale, and thin objects lose their supervision along with them.

We propose to diagnose each failure with its own cheap probe applied to the teacher, and to repair the pseudo-label rather than only reject it. The modality probe runs the teacher on complementary masked views: patches hidden in RGB are shown in thermal, and the reverse. A pixel counts as reliable if it is labeled consistently whichever sensor supplied its patch. The tokenization probe re-runs the teacher with the patch grid shifted by sub-patch offsets. A pixel whose label flips when the grid moves is a tokenization artifact. Averaging predictions over the shifted grids also gives a pseudo-label sharper than the patch size. The claim is that separating these two error sources yields more correct pseudo-labels at a given precision, and in particular more correct boundary and night-time labels, than filters that score uncertainty as a single number.

Core Claim

The research file points to two kinds of noise that have not been addressed together. In detection, Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification shows that pseudo-label noise clusters near decision thresholds and handles it with soft, uncertainty-weighted targets. TSP-Transformer and EchoONE both describe global attention as producing blurred boundaries, and the file attributes the blur in dense prediction to patch tokenization. These works motivate the proposal: they establish that boundary error is structural, not random. Within multimodal self-training, Towards Semi-supervised Dual-modal Semantic Segmentation (PD-Net) prunes 3D pseudo-labels that disagree with projected 2D predictions. CrossMatch uses a depth-aware auxiliary stream as an online teacher, together with prototype-based downweighting. Both rely on disagreement between modalities. That signal is blind to errors the modalities share, such as tokenizer blur inside one fused backbone. It also has the failure the file flags: under strict consensus, a dark RGB stream vetoes valid thermal predictions.

The method takes its two components from existing work. The modality probe reuses the complementary masking of Complementary Random Masking for RGB-Thermal Semantic Segmentation (CRM). CRM masks complementary patches across RGB and thermal as a supervised training regularizer. We instead use the masking at teacher inference time as a reliability test, and to our knowledge no work in the file does this. The stability criterion comes from Leveraging Local Variance for Pseudo-Label Selection (LVM), which keeps a pseudo-label only if it is both confident and stable under small perturbations. We change which perturbations are used. LVM uses generic augmentations on single-modality classification. We use two structured perturbations, chosen so that each targets one known failure mechanism.

Let $p_{k,s}(x)$ be the teacher's class distribution at pixel $x$ under complementary-mask view $k \in \{1..K\}$ and grid offset $s \in \{1..S\}$. The pseudo-label is $\hat y(x) = \arg\max \bar p(x)$, where $\bar p$ is the average over all $K S$ views. The per-pixel loss weight is

$$w(x) = a_{\text{mod}}(x)\, a_{\text{grid}}(x),$$

where $a_{\text{mod}}(x)$ is the fraction of mask views whose argmax, averaged over offsets, equals $\hat y(x)$. Likewise $a_{\text{grid}}(x)$ is the fraction of offsets whose argmax, averaged over mask views, equals $\hat y(x)$. Keeping the two factors separate lets us report which failure each rejected pixel exhibits. It also prevents a boundary pixel from being discarded merely because one sensor is dark.

Research Program

The base model is CMX (CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers) with a MiT-B2 backbone. We use it because it is the public RGB-X cross-attention fuser in the file, and its fusion module is where both failure modes meet. Training follows a standard mean-teacher loop, in which the teacher is an exponential moving average of the student. Only the pseudo-label generation and weighting change, so any gain can be attributed to the probes.

The work proceeds in three phases. The first is a diagnosis study. MFNet is fully annotated, so we can hide labels and measure where pseudo-label errors actually fall. We split errors into those inside a thin band around ground-truth boundaries and those in object interiors, and further by day versus night. We then score how well each signal detects errors, using area under the ROC curve for separating wrong from right pseudo-labels. The signals compared are softmax confidence, Monte Carlo dropout variance, LVM-style augmentation variance, the modality probe, and the tokenization probe. The second phase trains with the combined weight $w(x)$ and the shift-averaged pseudo-labels. The third phase stress-tests a known weak point of cross-modal attention, misregistration between sensors, by artificially translating the thermal image by a few pixels. The question is whether the modality probe flags the misaligned region or instead lets erroneous thermal texture spread, a risk the file raises for dense cross-attention.

The default cost is K=2 complementary pairs (four masked views) times S=4 grid offsets, about 16 teacher forward passes per unlabeled batch. Every filtering baseline below receives the same number of teacher passes.

Baselines

The naive reference is CMX trained only on the labeled fraction. Next to it is CMX with a mean teacher and a fixed confidence threshold of 0.95, the FixMatch-style recipe a practitioner would try first. The file reports no semi-supervised RGB-T numbers. As fully supervised ceilings on MFNet we quote CMX at 59.7% mIoU (CMX) and CRM at 61.2% mIoU (CRM). Closing a stated fraction of the gap between the labeled-only model and these ceilings is the target.

The strongest competitor on the modality question is cross-modal consensus filtering in the style of PD-Net. We reimplement it on CMX: separate RGB-only and thermal-only heads, with a pixel kept if it is confident or the two heads agree. We also run a CrossMatch-style variant, in which a thermal-aware auxiliary stream provides consistency targets and class prototypes downweight outlying pixels. The file reports no result for either in RGB-T semi-supervised segmentation. PD-Net's 63.38% 3D mIoU on ScanNet with 20% labels and CrossMatch's 57.7% mIoU on GTA5→Cityscapes are in different settings and cannot be compared directly.

The strongest competitor on the boundary question is LVM-style variance filtering with standard augmentations, including random translations, at an equal pass budget. The second is Monte Carlo dropout reweighting as used in Push the Boundary of SAM. All methods share the backbone, labeled splits, schedule, and a matched hyperparameter search of at most 12 trials, tuned on a held-out labeled validation split and never on test.

The ablations isolate the key ingredients. These are the modality probe alone, the tokenization probe alone, the two probes with a single merged uncertainty score instead of separate factors, and probes used only for weighting versus also for shift-averaged relabeling. Beating the field means a statistically reliable gain over the best consensus and variance baselines on boundary F-score and night-split mIoU across three seeds, without losing overall mIoU.

Why not just…?

Why not just use cross-modal voting, as PD-Net does? Voting compares the two streams, so it cannot detect an error both streams share. Tokenizer blur inside a fused transformer is exactly such an error. Strict voting also discards correct thermal labels whenever RGB is blind, as the file notes. The diagnosis phase settles this directly: if voting's error-detection AUROC within the boundary band matches the tokenization probe's, the second probe adds nothing and we will say so.

Why not just measure variance under ordinary augmentations, as LVM does? This is the real threat, because a random translation partly mimics a grid shift. The difference is that grid offsets are chosen to cover all sub-patch phases, and their average is used as a higher-resolution label rather than only as a filter. We run LVM with translations at an equal number of passes. If it matches the tokenization probe, we will report that LVM with translation already solves the boundary half of the problem. The contribution then narrows to the modality probe and the decomposed weighting.

Why not just take pseudo-labels from SAM, or sharpen them with a CRF? Push the Boundary of SAM shows that SAM's zero-shot masks fail on blurry boundaries without correction. SAM-guided Pseudo Label Enhancement needed several filtering stages to make projected SAM masks usable. SAM also sees only RGB, so it is weakest exactly on night scenes, where thermal matters. We include a SAM-mask-refined baseline on the RGB channel to quantify this.

Why not just use a CNN or local adapters to remove the blur at its source? Local convolution adapters, as in MaPPER, reduce blur but do not remove it. The pseudo-label filter is still needed, and the two approaches combine rather than compete. We test the probes on a CMX variant with local convolution adapters added. If the tokenization probe's benefit disappears there, the honest conclusion is that architecture alone fixes the boundary problem.

Proposed Experiments

The main benchmark is MFNet, 1,569 RGB-T pairs, with 1/16, 1/8, and 1/4 labeled splits. Results are reported overall and separately for day and night, a split that the modality probe is designed to exploit. PST900 and KP test transfer to subterranean and large-scale driving scenes. NYU Depth V2 checks whether the approach carries over to RGB-D. Metrics are mIoU, boundary F-score within a narrow pixel band, IoU on thin classes, pseudo-label precision against held-out ground truth, and the density of correct pseudo-labels at fixed precision.

The decisive checks follow directly from the claim. First, boundary-band errors should be detected far better by the tokenization probe than by confidence or voting. Second, night-split errors should be detected better by the modality probe than by grid-shift variance. If the two probes are not specialized in this way, the decomposition is not real. Third, at matched pseudo-label precision, our method should keep more correct night and boundary pixels than PD-Net-style voting. Fourth, under synthetic thermal misregistration, $a_{\text{mod}}$ should drop in the shifted region, rather than misaligned thermal texture appearing in the pseudo-labels.