In machine learning, we often need to align two different datasets—for example, adapting a model trained on synthetic images to work on real-world photos. A common way to do this is through Optimal Transport (OT), a mathematical framework that finds the most efficient way to "move" one distribution to match another.
However, standard alignment techniques harbor a hidden assumption: they assume the relative frequency of classes is the same in both datasets. When this assumption fails—a phenomenon known as target shift or class weight bias—traditional alignment methods often collapse, forcing incorrect matches between unrelated categories.
The Mass Conservation Trap
The fundamental problem lies in the constraint of mass conservation. In standard OT, every "unit of mass" in the source distribution must be mapped to the target distribution. As Redko et al. explain, this forces the model to satisfy global marginal distribution requirements at the expense of semantic accuracy.
Imagine a source dataset that is 90% "cats" and 10% "dogs," while the target dataset is 50% "cats" and 50% "dogs." To satisfy the requirement that all source samples must be mapped, a standard OT algorithm is forced to map the "excess" source cats to target dogs. Chuang et al. describe this as the algorithm being "distracted" by imbalanced classes, leading to majority class samples being spread across multiple minority classes in the target domain.
This issue isn't limited to OT. Yan et al. note that even Maximum Mean Discrepancy (MMD)—a popular alternative for measuring distribution distance—fails when class priors shift, as the metric attempts to force-match mismatched proportions.
Relaxing the Constraints: Unbalanced Optimal Transport
To solve this, researchers have moved toward Unbalanced Optimal Transport (UOT). Instead of requiring a strict mapping of all data, UOT relaxes the marginal constraints, allowing the model to "ignore" excess mass or outliers that don't have a clear correspondence in the other domain.
Tran et al. demonstrate that by relaxing these constraints, models become significantly more robust to outliers and class imbalances in complex settings like single-cell genomics. Similarly, Lee et al. apply this intuition to 3D point clouds, showing that UOT prevents the distribution shifts that occur when category distributions differ significantly between partial and complete shapes.
Because UOT can be computationally expensive, Gazdieva et al. recently introduced U-LightOT, a lightweight solver that makes these unbalanced alignments feasible for large-scale problems, solving complex entropic transport tasks in minutes on standard hardware.
Re-weighting and Structural Alignment
Beyond relaxing mass constraints, other strategies focus on adjusting the importance of specific samples or aligning data at a higher level of abstraction.
- Weighted Discrepancies: Yan et al. proposed Weighted MMD (WMMD), which re-weights source samples to match the estimated class priors of the target domain. This ensures that the alignment process prioritizes matching the classes in their correct proportions.
- Cluster-Level Matching: Rather than matching individual instances, Liu et al. suggest aligning cluster centers. By matching clusters rather than every single point, their method bypasses the "class imbalance trap" where excess source samples generate inaccurate pseudo-labels for the target.
- Structural Regularization: Rahbar et al. introduced a Sum-of-Norms (SON) regularization framework. This approach encourages the transport mechanism to respect the underlying class structure, identifying superfluous data rather than forcing erroneous mappings between unrelated classes.
Applications in Noisy and Shifting Environments
The need for imbalance-sensitive alignment extends to various domains where data is rarely clean or balanced:
- Label Denoising: In datasets with high noise, Chang et al. and Pan et al. argue that standard OT assumes uniform class distributions, which fails on real-world "long-tailed" data. By making the transport process structure-aware or combining it with robust loss functions, models can better filter out incorrect labels even when some classes are rare.
- Time Series and Geophysics: Painblanc et al. highlight that in time series adaptation (e.g., satellite monitoring across different regions), class proportions often shift due to geography. Their Match-And-Deform framework uses OT to handle these shifts while simultaneously aligning temporal axes.
- Synthetic Data Generation: Even when creating synthetic datasets, Fan and Alvarez-Melis show that if the source datasets have imbalanced classes, the resulting synthetic "blend" will be degraded unless the alignment mechanism specifically accounts for those imbalances.
By moving away from the rigid requirement of global mass conservation, these methods allow models to find semantically meaningful correspondences in the messy, imbalanced reality of real-world data.
Key Foundational Work
Mind the Class Weight Bias: Weighted Maximum Mean Discrepancy for Unsupervised Domain Adaptation (2017) — contributed 1 concept elements to this direction.
Optimal Transport for Multi-source Domain Adaptation under Target Shift (2019) — contributed 1 concept elements to this direction.