Robust Multi-View Fusion via Prototype-Anchored Unbalanced Optimal Transport

Zhang Han, Xingwen Zhao, Hui LiICML 2026 · OpenReview

Introduction

In multi-modal AI—such as systems recognizing human actions via RGB video, skeletal data, and infrared sensors—the standard operating principle has long been: "If a view is observed, use it." However, real-world data is inherently messy. Sensors fail, cameras suffer occlusion, and tracking algorithms glitch.

Traditional fusion methods, such as attention mechanisms or simple pooling, are "balanced." They attempt to explain every piece of input they receive. Consequently, if one view is heavily corrupted, it can pull the entire fused representation toward the noise, leading to misclassification. This paper identifies a fundamental flaw in this approach: strict mass conservation is a poor constraint when a portion of that "mass" is unreliable. To address this, the authors propose a novel multi-view fusion module based on Unbalanced Optimal Transport (UOT), allowing the model to dynamically "trim" unreliable evidence through a differentiable penalty mechanism.

Methodology: Prototype-Anchored UOT

The proposed architecture introduces a Prototype-Anchored Unbalanced Optimal Transport module designed to fuse multiple modalities robustly. The pipeline consists of four main steps:

  1. Atom Extraction: Each observed view (e.g., RGB or depth) is compressed into a small set of "latent atoms" using a learned-query pooling mechanism, conceptually similar to the Perceiver architecture.
  2. Shared Prototypes: The model maintains a set of learnable "prototypes" within a shared latent space. These prototypes act as universal anchors for the fusion process.
  3. Unbalanced Optimal Transport (UOT): Instead of forcing a perfect match between view atoms and prototypes, the module employs UOT with a KL-relaxed objective. In standard Optimal Transport, all "mass" must be moved from the source to the destination. In Unbalanced OT, the model can opt to leave some mass unmatched if the geometric cost (distance in the latent space) is too high. It simply pays a bounded penalty, controlled by a hyperparameter $\tau$, to ignore that data.
  4. Differentiable Solver: The fusion objective is solved using unrolled Sinkhorn iterations, making the entire trimming and matching process end-to-end trainable via standard backpropagation.

Key Insights

  • Trimming without Thresholds: Traditional denoising often relies on hand-tuned thresholds (e.g., ignoring data with confidence scores below 0.5). UOT provides a naturally differentiable alternative, discarding outliers based on their geometric compatibility with the learned prototypes.
  • Bounded Penalty: A core theoretical contribution of the paper is proving that the cost of discarding "bad" atoms is strictly bounded by their mass, independent of how far away they are in the feature space. This mathematical guarantee prevents a single extreme outlier from corrupting the entire fused representation.
  • The Power of "Less is More": The research demonstrates that in contaminated environments, utilizing less of an observed view often yields a more reliable classifier.

Experimental Results

The method was rigorously evaluated on two large-scale action recognition benchmarks: NTU RGB+D 60 (56,880 clips) and NTU RGB+D 120 (114,480 clips), utilizing RGB, skeleton, and infrared modalities. The authors simulated severe missingness and feature-space corruption to stress-test the module.

  • Superior Robustness: On the NTU 60 dataset under an extreme 80% missing-view rate, the proposed UOT method achieved 78.3% accuracy, outperforming the strongest recent baseline (RCMCL) by a substantial 7.5 percentage points.
  • Resilience to Missing-Rate Shift: When trained on data with a 20% missing rate but tested on an 80% missing rate, the model's performance dropped by only 15.1 points. In contrast, standard Late Fusion plummeted by 31.2 points under the same shift.
  • Graceful Degradation: When RGB features were intentionally corrupted with Gaussian noise, the UOT module automatically reduced the "transported mass" of the RGB view. It effectively "turned down the volume" on the noisy sensor to preserve overall accuracy.
  • High Efficiency: Because the module matches a small number of atoms (e.g., 16) to a small number of prototypes (e.g., 16), it adds only about 11% wall-clock overhead during inference, a negligible cost compared to heavy backbone encoders like Video Swin Transformers.

Limitations and Future Directions

While highly effective, the UOT fusion approach has limitations. Its "trimming" mechanism relies on geometric dependency; if an encoder maps a corrupted input directly onto a valid prototype, the system will be confidently wrong. Furthermore, the model is vulnerable to systematic bias—if all sensors fail in the same direction, the model lacks a "clean" anchor and will fail. Finally, there is a risk that rare but correct patterns (e.g., unusual camera angles) might be treated as outliers and trimmed, potentially degrading performance on long-tail data distributions.

Ultimately, while Prototype-Anchored UOT is a powerful architectural tool for robustness, it judges reliability purely through learned geometry and should be paired with broader safety mechanisms in critical deployments.

Summary updated

Abstract

Multi-view classifiers typically fuse all observed views into a single representation, which becomes fragile when some views are missing or corrupted.We propose a prototype-anchored fusion module based on an entropically regularized unbalanced optimal transport (UOT) barycenter.Each view is summarized into a small set of learned atoms and is matched to a shared prototype support; fusion outputs a probability measure over prototypes with fixed dimension.By relaxing marginal constraints with a generalized KL penalty, the UOT objective can leave a fraction of view mass unmatched when matching is geometrically costly, yielding a simple differentiable trimming mechanism without hand-tuned thresholds.We provide a basic theoretical result showing that discarding an arbitrary subset of atom mass incurs a penalty bounded by its total mass, independent of transport distances.Experiments on multi-view action recognition benchmarks under simulated missing views, missing-rate shift, and feature-space corruption demonstrate consistently improved stability under severe missingness with modest overhead on top of strong backbones.
BibTeX
@inproceedings{han2026robust,
  author = {Zhang Han and Xingwen Zhao and HUI LI},
  title = {{Robust Multi-View Fusion via Prototype-Anchored Unbalanced Optimal Transport}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=d9obyVpqAu},
}
Figure