Masked autoencoders learn visual features by hiding most patches of an image and asking a Vision Transformer (a network that splits an image into a grid of small square patches and lets every patch attend to every other) to fill in what is missing. MultiMAE extends this from color pixels to several "modalities" at once: depth and semantic segmentation maps that do not exist for ImageNet are generated by off-the-shelf teacher networks and then masked and reconstructed alongside RGB. These pseudo-labels are wrong in predictable places, such as object boundaries, reflective and dark surfaces, distant depth, and rare classes. Yet the student is asked to fit every pseudo-labeled patch with equal weight from the first step of training. A separate body of work on loss warm-up shows that fitting hard or unreliable targets before the network has usable features pushes it into poor solutions.
We propose to warm up each patch separately. Training starts with color reconstruction, which is clean. A pseudo-label patch is admitted into the loss only once the student can already predict it reasonably well from context, and the pace at which harder patches are admitted is tied to a measurable plateau in the color reconstruction loss rather than to a hand-set epoch count. The bet is that pseudo-label noise is spatially concentrated. If it is, a single global warm-up must either withhold the useful clean patches early or let the noisy ones in. A per-patch gate can do neither: it delivers clean signal immediately and defers noisy signal until features are strong enough to absorb it, or excludes it entirely.
Core Claim
MultiMAE pretrains a shared ViT on RGB, pseudo-depth from DPT, and pseudo-segmentation from Mask2Former. It samples visible tokens by a Dirichlet split across modalities and uses static reconstruction losses throughout. Our claim is that replacing this static objective with a student-paced, patch-level admission rule improves downstream dense transfer. The test is NYUD-v2 segmentation and depth plus the multi-task suite. We further claim that the gain comes specifically from where the gate admits signal, not from simply having less pseudo-label loss early.
The proposal is falsifiable in a direct way. If a tuned global linear ramp on the pseudo-label losses matches it, the right conclusion is that warm-up matters but per-patch adaptivity does not, and we will report it that way.
Research Program
The method keeps MultiMAE's architecture, masking, and decoders unchanged. Only the loss on masked depth and segmentation patches changes. Let $\ell_{p,m}$ be the student's reconstruction loss on masked patch $p$ of pseudo-modality $m$ at the current step. The weight on that patch is
$$ w_{p,m}(t) = \sigma\!\left(\frac{\lambda_m(t) - \ell_{p,m}}{T}\right), $$
where $\sigma$ is the logistic function and $T$ is a temperature. The threshold $\lambda_m(t)$ is the $q(t)$-quantile of $\ell_{\cdot,m}$ over a running buffer of recent patches. The weights are detached from the gradient so the student cannot lower its loss by refusing patches.
The admitted fraction $q(t)$ starts at a small value $q_0$ and rises toward one in increments. The trigger is internal: an increment fires when the exponential moving average of the RGB reconstruction loss changes by less than a fixed relative amount over a window. This replaces the fixed epoch counts that the warm-up literature relies on with a signal from the model's own progress, which is the "heuristic phase transition" problem flagged across that literature.
The design borrows from three warm-up papers, each of which motivates a different part:
- Anchor Loss modulates each sample's penalty by a prediction-based difficulty anchor. Its authors require a cross-entropy warm-up because the anchor is noisy early. We use the same student-side difficulty signal, but apply it per spatial patch rather than per image.
- Asymmetric Loss reports that prediction-derived signals are too noisy to steer training at initialization. This is why our gate is a quantile within a buffer rather than an absolute threshold, and why RGB-only training precedes it.
- To Smooth or Not? shows that sharpening targets on noisy labels before features form causes premature convergence. This motivates deferring the hard pseudo-label patches.
None of these works operate on dense, masked, multi-modal pretraining.
The analysis half of the program asks whether the gate does what we say. We will run DPT and Mask2Former on NYUD-v2 training images, where ground-truth depth and segmentation exist (Silberman et al.). This gives a per-patch pseudo-label error. We then measure whether the patches the gate admits late, or never, are the ones the teachers got wrong. The same machinery tells us how concentrated pseudo-label noise actually is. If that noise is spread uniformly, the core premise fails before any downstream number is computed.
Patch-based self-attention plays the role of the constraint that makes the idea natural. The loss already decomposes over patch tokens, so gating costs one sort over a buffer and adds no forward passes.
Baselines
The naive reference point is RGB-only masked autoencoding with the same ViT-B encoder, schedule, and fine-tuning recipe. It tells us how much any pseudo-labeled pretraining buys. The strongest existing method is MultiMAE as published, with static losses. The file reports that its NYUv2 semantic segmentation reaches 52.0 mIoU when fine-tuned on RGB alone and 56.0 mIoU when depth is also used, along with 83.3% ImageNet-1K top-1 from RGB input (MultiMAE). No reported result was found on Lacuna for MultiMAE's NYUv2 depth error.
Our full budget is unlikely to equal the original, so we will reimplement MultiMAE inside our own pipeline and compare against that reproduction. All arms share the same pseudo-labels, masking, pretraining epochs, and fine-tuning hyperparameter sweep. Beating MultiMAE means a gain on NYUv2 segmentation in both the RGB-only and RGB+depth fine-tuning settings that exceeds the spread across three seeds, together with no loss on ImageNet top-1.
The global warm-up baseline uses the same RGB-first start. It then ramps the depth and segmentation loss weights linearly from zero to one over a ramp length chosen from a sweep. The sweep gets the same tuning budget as our two main hyperparameters, $q_0$ and the plateau tolerance. This is the most important comparison, because it is the simplest version of "warm-up."
A teacher-confidence filter drops pseudo-label patches using the teacher's own uncertainty: Mask2Former's per-pixel class confidence, and DPT's horizontal-flip disagreement for depth. The filter is tuned to the same admitted fraction.
Two ablations isolate the key ingredient. The first, a random-gate ablation, admits the same number of patches per step as our gate but chooses them uniformly. If it matches the full method, then location does not matter. The second, a fixed-schedule ablation, keeps the per-patch gate but advances $q(t)$ on a hand-set epoch schedule, which isolates the plateau trigger.
As a gradient-side competitor, we will run RotoGrad on the pooled encoder output during pretraining, using the subspace rotation its authors propose for high dimensions. RotoGrad rescales and rotates each task's gradients to remove magnitude and direction conflicts. The file reports qualitative NYUv2 gains for it but gives no reported number on Lacuna.
Why not just…?
Why not just ramp up the pseudo-label losses globally? This is the real threat. A linear ramp captures the "warm-up first" lesson from Outlier Exposure with confidence control and the predict-then-optimize work, and it costs nothing. Our argument against it is purely spatial: a global ramp cannot separate a correct floor-depth patch from a wrong mirror-depth patch in the same image. That argument only holds if pseudo-label error is spatially concentrated. The NYUD-v2 teacher-error analysis tests this directly, and the matched-budget comparison settles it downstream. If the ramp ties, we will drop the per-patch claim.
Why not just filter pseudo-labels by the teacher's own confidence? Teacher confidence is static and miscalibrated in exactly the regions that matter, because confident mistakes are the dangerous ones. It also says nothing about whether the student has features ready to use a correct but hard target. Our gate uses the student's state, which changes over training. The confidence-filter baseline at a matched admitted fraction measures whether this distinction matters. If the filter wins, the simpler method should be adopted.
Why not just pretrain RGB-only MAE and then fine-tune on the dense tasks? The file's sequential-versus-joint evidence comes mostly from other domains (Wang et al., Situ et al.), so we do not lean on it. The direct evidence is MultiMAE's own gain over RGB-only pretraining on dense transfer. The RGB-only baseline is in our table, so this alternative is measured rather than argued away.
Why not just fix the conflict with gradient surgery such as RotoGrad? RotoGrad addresses conflicts between task gradients. Our target is different: individual targets that are wrong. Rotating a gradient computed from a bad depth label still fits the bad label. Running RotoGrad as a baseline, and in combination with our gate, shows whether the two problems overlap.
Proposed Experiments
Pretraining uses ImageNet-1K with MultiMAE's DPT and Mask2Former pseudo-labels and a ViT-B encoder. All arms first run at a reduced, equal epoch budget with three seeds. The best two configurations are then rerun at a longer budget.
Transfer is evaluated in three ways. First, NYUD-v2 semantic segmentation in mIoU, with RGB-only and RGB+depth fine-tuning, following MultiMAE. Second, NYUD-v2 depth, by RMSE. Third, four-task NYUD-v2 and five-task PASCAL-Context, each summarized by average relative improvement over single-task models, using a fixed DeMT-style decoder. For orientation, DeMT reports 3.36% on NYUD-v2 with a Swin-T backbone. That figure uses a different backbone, so we cite it only as a reference scale and not as a head-to-head target. ImageNet top-1 is a guard against harming classification. Boundary detection gets separate reporting, because ATRC found that edge targets conflict most with smooth dense targets. If our gate helps anywhere, it should help there.
Three outcomes are decisive. The first is the gate-placement analysis: on NYUD-v2, the rank correlation between when the gate admits a patch and the teacher's true error on that patch should be clearly positive. The second is that the full method should beat both the tuned global ramp and the random-gate ablation on NYUv2 segmentation and on the multi-task average. The third is that the plateau trigger should match or beat the best hand-set schedule without per-dataset tuning.
Failure on the first check stops the project. Failure on the second, with success on the first, would mean the gate finds noise but the noise does not hurt. That finding would itself argue that MultiMAE's static losses are adequate.
References
- MultiMAE: Multi-modal Multi-task Masked Autoencoders
- Anchor Loss: Modulating Loss Scale based on Prediction Difficulty
- Asymmetric Loss For Multi-Label Classification
- To Smooth or Not? When Label Smoothing Meets Noisy Labels
- Indoor Segmentation and Support Inference from RGBD Images
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Outlier exposure with confidence control for out-of-distribution detection
- Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
- Assisting Main Task Learning by Heterogeneous Auxiliary Tasks with Applications to Skin Cancer Screening
- The Role of Context for Object Detection and Semantic Segmentation in the Wild
- DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction
- Exploring Relational Context for Multi-Task Dense Prediction