Checking Pseudo-Labels by Walking the Bridge Between Modalities

When a model must match very different kinds of data, such as daytime color photos of people against nighttime thermal images, a common trick is to build an in-between "bridge" representation. The bridge blends features from both sides, so the model learns two small jumps instead of one large one. In the semi-supervised version of this problem, one side is labeled and the other is not. The unlabeled side gets labels guessed by the model itself, called pseudo-labels. The bridge is then built by mixing a labeled sample with an unlabeled sample that shares its pseudo-label. When that guess is wrong, the bridge is built from two different people. The model is then trained to treat them as one person, which reinforces its own mistake.

This note proposes to turn that weakness into a test. For a correct pseudo-label, the interpolated features between the labeled anchor and the unlabeled sample should stay inside one class region along the whole path. For a wrong pseudo-label, the path must pass through the true class or through a boundary region. Reading the classifier along a handful of interpolation points gives a pseudo-label verifier. It costs a few passes through the classifier head, not extra backbone passes, a generative translator, or a teacher ensemble. The same verifier then decides which samples may enter the bridge and how much each one counts in the loss.

Core Claim

Cross-modal pseudo-label errors can be detected more reliably, per unit of compute, by checking whether the prediction stays consistent along the feature path from a labeled anchor to the unlabeled sample than by checking the unlabeled sample's own confidence or loss. Gating the bridge with this signal should improve semi-supervised cross-modal retrieval while adding little training cost.

Concretely, let $h_s$ be the feature of a labeled visible image of class $\hat y$, and let $h_t$ be the feature of an unlabeled infrared image pseudo-labeled $\hat y$. Let $\hat c$ be the model's second-choice class for $h_t$. For mixing weights $\lambda \in \{0, 0.25, 0.5, 0.75, 1\}$, define the bridge point $h(\lambda) = (1-\lambda)h_s + \lambda h_t$. Define the path score as

$$S(x_t) = \min_{\lambda} \big[\, p(\hat y \mid h(\lambda)) - p(\hat c \mid h(\lambda)) \,\big],$$

where $p(\cdot \mid h)$ is the classifier's probability for a class given a feature. $S$ is averaged over a few anchors of class $\hat y$. A high $S$ means the path never leaves the region of $\hat y$. A low or negative $S$ means the path crosses a region where another class takes over.

This differs from plain confidence in two ways. Every path starts at a ground-truth-labeled anchor. And the score probes the space between the two samples, which is exactly where the bridge will later put training signal.

Where This Sits in Prior Work

The closest work motivates the gap rather than filling it. MUCG does semi-supervised visible-infrared re-identification (matching the same person across cameras and modalities). It mixes visible and infrared feature maps to form a dynamic intermediate modality. It then downweights infrared pseudo-labels using the "memory effect", the observation that networks fit clean labels before noisy ones, and it aligns per-identity class centers. However, its mixing step is not conditioned on each sample's reliability. We keep its setting and make the bridge itself the reliability check. FIVE mixes 2D and 3D features that share a pseudo-label and fights noise with votes from several prototypes per class and a sharpness-aware objective (one that seeks flat minima). The prototype votes judge the unlabeled sample's position, not the segment it will be mixed along. The sharpness-aware step roughly doubles gradient cost. PRAISE translates features across modalities with a CycleGAN-style network and fits a Beta mixture to losses to flag noisy clusters. Its authors name the three-stage pipeline's cost and instability as open problems, which constrains us to stay non-generative and end-to-end.

The bridging method itself comes from lightweight, non-generative intermediate modalities. PaRe uses a one-layer gate to replace target patches with source patches on a decaying schedule and shows this smooths the loss landscape. It is fully supervised and never faces wrong labels. The X Modality and privileged intermediate information show that cheap training-time intermediates can be discarded at inference. The cost constraint comes from the synthetic-data literature. DISCO shows that filtered data beats raw volume, but its teacher filtering is paid per discarded sample. UDAPDR replaces filtering with ensembles of heavy rerankers. The diffusion broker modality had to be distilled before it was practical. Our position is that verification should be paid in forward passes of a classifier head, and we will report it that way.

Research Program

We will first build the verifier as a drop-in module on the MUCG training recipe. Mixing happens at the final pooled feature, so each path point costs only a pass through the classifier head. As an ablated variant, mixing also happens at the intermediate layer that DIMG uses, which costs partial backbone passes. Each epoch, every pseudo-labeled infrared sample gets a path score from three anchors and five mixing points.

The score is used in two places. First, as a gate: only samples with $S$ above a running per-class quantile enter bridge mixing. Using a quantile rather than a fixed threshold addresses the threshold brittleness noted in Combating Noise. Second, as a soft weight $w = \sigma(S/T)$ on the identification loss, where $\sigma$ is the logistic function and $T$ is a temperature. This replaces the memory-effect weight.

Next, we will study the analysis question that none of the prior papers answer: how much does a wrong pseudo-label damage the bridge? On the benchmarks, infrared identity labels exist but are hidden during training. This lets us measure pseudo-label error rates, noise-detection accuracy, and the purity of mixed samples at every epoch. It also lets us inject controlled label flips.

Finally, we port the module unchanged to FIVE's 2D-3D setting to test whether the idea carries beyond RGB-infrared.

A risk we take seriously is that the verifier uses the model's own classifier and could inherit its blind spots. The labeled anchor is the only external grounding. We will measure detection accuracy at different stages of training, because Tri-net and Self-Ensembling Attention Networks indicate that early errors do the most damage.

Baselines

The naive reference is MUCG's backbone trained with hard, confidence-thresholded infrared pseudo-labels and ungated feature mixing. This shows what the bridge does when fed unchecked labels.

The strongest existing method on the main question is MUCG itself. It is the only method in the file that combines a feature-mixing bridge with pseudo-label weighting in this exact setting. It reports Rank-1 68.8% and mAP 65.9% on SYSU-MM01, Rank-1 86.9% and mAP 76.7% on RegDB, and Rank-1 51.9% on LLCM, per MUCG. We will rerun it from the same backbone, data splits and label budget, with equal hyperparameter search budgets for its loss weights and ours. Beating it means a consistent Rank-1 and mAP gain on all three datasets across at least three seeds, with total training GPU-hours within about ten percent of MUCG's.

For the 2D-3D port, the reference is FIVE. It reports 72.94% 2D-to-3D mAP on ModelNet40 with 800 labeled samples, per FIVE. We will compare two configurations: path gating with sharpness-aware minimization disabled, and path gating on top of full FIVE. The first tests whether a cheap verifier can replace the costly optimizer. The second tests whether the two are complementary.

For the verifier alone, we compare against four pseudo-label reliability signals under a matched budget of forward computation:

  • softmax confidence;
  • memory-effect loss weighting, as in MUCG and PRAISE's Beta mixture;
  • dropout-stability consensus with as many full passes as our budget allows, following Tri-net;
  • multi-prototype voting, following FIVE.

The measure is noise-detection AUROC against the hidden infrared labels. The key ablations isolate the ingredient we claim matters. A single midpoint mix ($\lambda = 0.5$ only) tests whether the whole path matters. A path to a random same-modality sample instead of a labeled visible anchor tests whether anchoring matters. We also separate gating-only from weighting-only.

Why not just…?

Why not just threshold the infrared sample's confidence before mixing? This is the most obvious fix and would cost nothing. Combating Noise shows that errors cluster near decision boundaries, where confidence is least informative. Cross-modal samples are exactly where networks are overconfident. Still, the fix could plausibly work, so the noise-detection AUROC comparison at matched compute settles it directly. If confidence matches the path score, the proposal reduces to an analysis paper about bridge corruption, and we will report it as such.

Why not just use MUCG's memory-effect weighting, which already exists? It is a real threat because it is free. It judges each sample by how fast its loss falls, which says nothing about whether mixing it with a particular anchor is safe. It also inherits early-training immaturity. We test it head-to-head as a detector and as a full training signal. We also try combining it with the path score, in case the two signals are complementary.

Why not just translate one modality into the other with a generator? PRAISE did this and reports instability and three-stage complexity. The broker-modality work had to distill diffusion models to make them usable. A generator also does not tell you which pseudo-labels are wrong. We do not rerun a generative pipeline and cite its reported limitations instead. The PRAISE numbers (for example, SYSU-MM01 all-search Rank-1 59.44%) come from the unsupervised setting and are not directly comparable to ours.

Why not just rely on sharpness-aware training to tolerate the noise? FIVE's results suggest it helps. It tolerates noise rather than removing it, at roughly double the gradient cost. The FIVE-without-sharpness-aware-minimization configuration shows whether cheap gating recovers that gain.

Proposed Experiments and Decisive Checks

The first decisive check is detection. On SYSU-MM01, RegDB and LLCM, we log the true pseudo-label error rate each epoch. The path score must reach higher AUROC than every reliability signal in the Baselines section at the same number of forward passes, and this must hold in the first quarter of training, not only once the model has matured. If it fails here, the method does not proceed to the downstream comparison.

The second check is controlled corruption. We flip 10%, 20% and 40% of infrared pseudo-labels and measure three things: the purity of mixed samples, the downstream Rank-1 and mAP, and how fast the error rate grows over rounds. Ungated mixing should degrade faster than gated mixing, which would give the first direct evidence that noisy labels poison the bridge. If ungated mixing is no worse, the premise of the gap is wrong and we will say so.

The third check is the end-to-end comparison against MUCG and FIVE. We report the metrics and baseline numbers above plus cost accounting: total GPU-hours, forward passes per pseudo-label accepted, and wall-clock time per epoch. This puts the verification cost in the same terms that the synthetic-data work uses for filtering.

As a stretch goal, we withhold 90% of NinaPro labels in PaRe's patch-replacement setting and apply path gating to its patch curriculum. The file contains no reported result for semi-supervised NinaPro. PaRe's supervised error rate of 6.53% serves only as an upper reference.