Abstract
Machine learning models often struggle with out-of-distribution (OOD) generalization, failing when test data deviates from training distributions. We present Topology-aware Robust Optimization (TRO), a framework that learns the underlying manifold of data distributions to identify structurally "influential" training groups. By leveraging physical priors or data-driven diffusion metrics, TRO constrains the uncertainty set in robust optimization, focusing on groups that act as bridges to unseen data. TRO achieves state-of-the-art performance across classification, regression, and segmentation tasks, outperforming traditional Distributionally Robust Optimization (DRO) by avoiding over-optimization on irrelevant outliers.
Introduction: The OOD Challenge
Standard machine learning models typically rely on Empirical Risk Minimization (ERM), which assumes that training and test data are independent and identically distributed (i.i.d.). In real-world scenarios, this assumption rarely holds. While Distributionally Robust Optimization (DRO) attempts to mitigate this by minimizing the "worst-case" risk across potential distributions, it often becomes overly pessimistic. DRO may focus excessively on unrealistic outliers or "worst-case" training groups that bear little resemblance to the actual test environment.
The authors of TRO argue that data distributions are not isolated points but rather lie on a structured manifold. By uncovering this topology, models can be guided toward more plausible unseen data rather than being distracted by noise.
The TRO Framework
TRO operates in two distinct phases: Topology Learning and Learning on Topology.
1. Topology Learning
The framework first constructs a graph where each node represents a data distribution (a "group") and edges represent their similarity. This can be achieved through two methods:
- Physical-based: Utilizing domain knowledge, such as geographic adjacency for weather stations or sensor networks.
- Data-driven: In the absence of priors, TRO employs Multiscale Diffusion Earth Mover’s Distance (EMD). This approach simulates a diffusion process over the data to measure connectivity between distributions, effectively mapping the manifold structure.
2. Learning on Topology
Once the graph is established, TRO calculates group centrality (specifically betweenness centrality). This metric identifies "influential" groups—those that serve as hubs or bridges within the topology. The optimization then solves a minimax problem, seeking a model that performs well on a weighted mixture of training groups. These weights are constrained by a "topological prior" derived from centrality scores, forcing the model to prioritize structurally important groups over irrelevant outliers.
Experimental Evaluation
The authors evaluated TRO across a diverse range of benchmarks:
- Synthetic Classification (DG-15/DG-60): Testing shifts in decision boundaries.
- Real-world Regression (TPT-48): Predicting monthly temperatures across 48 U.S. states, generalizing across geographic regions.
- Semantic Segmentation (Sen1Floods11): Global flood mapping using satellite imagery across 11 distinct events.
- DomainBed Benchmarks: Standard OOD tests including PACS, Terra Incognita, and VLCS.
Key Results
- Classification: On the DG-15 dataset, TRO achieved 67.89% accuracy, significantly outperforming ERM (58.00%) and standard DRO (43.22%).
- Regression: TRO consistently yielded the lowest Mean Squared Error (MSE) on the TPT-48 task, particularly for groups furthest from the training set.
- Segmentation: On Sen1Floods11, TRO reached a test IoU of 0.450, surpassing baselines like IRM and REx.
- SOTA on PACS: TRO set a new state-of-the-art average accuracy of 86.5%.
Discussion: Why Topology Matters
A critical insight from this work is that the "worst-case" group is not necessarily the most "influential" group. Standard DRO can fail by over-optimizing for an outlier that has no bearing on the test distribution. In contrast, TRO's use of centrality identifies groups that are better proxies for unseen data.
The data-driven topology often aligns with human intuition. For instance, in flood mapping, TRO identifies heavy-rainfall events as more central and influential than rare edge cases like dam collapses. Furthermore, by constraining the uncertainty set via graph structure, TRO provides tighter generalization bounds than traditional robust methods.
Limitations and Future Work
While effective, TRO faces challenges regarding computational cost, as data-driven topology learning requires calculating distances between all distributions. Currently, topology learning and optimization are separate steps; the authors suggest that developing an end-to-end differentiable pipeline is a vital next step. Additionally, the benefits of TRO are less pronounced in scenarios with very small distributional shifts, where group centralities tend to converge.
Summary updated