Topology-aware Robust Optimization for Out-of-distribution Generalization

Fengchun Qiao, Xi Peng

Abstract

Machine learning models often struggle with out-of-distribution (OOD) generalization, failing when test data deviates from training distributions. We present Topology-aware Robust Optimization (TRO), a framework that learns the underlying manifold of data distributions to identify structurally "influential" training groups. By leveraging physical priors or data-driven diffusion metrics, TRO constrains the uncertainty set in robust optimization, focusing on groups that act as bridges to unseen data. TRO achieves state-of-the-art performance across classification, regression, and segmentation tasks, outperforming traditional Distributionally Robust Optimization (DRO) by avoiding over-optimization on irrelevant outliers.

Introduction: The OOD Challenge

Standard machine learning models typically rely on Empirical Risk Minimization (ERM), which assumes that training and test data are independent and identically distributed (i.i.d.). In real-world scenarios, this assumption rarely holds. While Distributionally Robust Optimization (DRO) attempts to mitigate this by minimizing the "worst-case" risk across potential distributions, it often becomes overly pessimistic. DRO may focus excessively on unrealistic outliers or "worst-case" training groups that bear little resemblance to the actual test environment.

The authors of TRO argue that data distributions are not isolated points but rather lie on a structured manifold. By uncovering this topology, models can be guided toward more plausible unseen data rather than being distracted by noise.

The TRO Framework

TRO operates in two distinct phases: Topology Learning and Learning on Topology.

The TRO framework: transitioning from distributed datasets to physical or data-driven graphs, followed by multiscale diffusion density estimation and robust optimization.

1. Topology Learning

The framework first constructs a graph where each node represents a data distribution (a "group") and edges represent their similarity. This can be achieved through two methods:

  • Physical-based: Utilizing domain knowledge, such as geographic adjacency for weather stations or sensor networks.
  • Data-driven: In the absence of priors, TRO employs Multiscale Diffusion Earth Mover’s Distance (EMD). This approach simulates a diffusion process over the data to measure connectivity between distributions, effectively mapping the manifold structure.

2. Learning on Topology

Once the graph is established, TRO calculates group centrality (specifically betweenness centrality). This metric identifies "influential" groups—those that serve as hubs or bridges within the topology. The optimization then solves a minimax problem, seeking a model that performs well on a weighted mixture of training groups. These weights are constrained by a "topological prior" derived from centrality scores, forcing the model to prioritize structurally important groups over irrelevant outliers.

Experimental Evaluation

The authors evaluated TRO across a diverse range of benchmarks:

  • Synthetic Classification (DG-15/DG-60): Testing shifts in decision boundaries.
  • Real-world Regression (TPT-48): Predicting monthly temperatures across 48 U.S. states, generalizing across geographic regions.
  • Semantic Segmentation (Sen1Floods11): Global flood mapping using satellite imagery across 11 distinct events.
  • DomainBed Benchmarks: Standard OOD tests including PACS, Terra Incognita, and VLCS.

Key Results

  • Classification: On the DG-15 dataset, TRO achieved 67.89% accuracy, significantly outperforming ERM (58.00%) and standard DRO (43.22%).
  • Regression: TRO consistently yielded the lowest Mean Squared Error (MSE) on the TPT-48 task, particularly for groups furthest from the training set.
  • Segmentation: On Sen1Floods11, TRO reached a test IoU of 0.450, surpassing baselines like IRM and REx.
  • SOTA on PACS: TRO set a new state-of-the-art average accuracy of 86.5%.

Discussion: Why Topology Matters

A critical insight from this work is that the "worst-case" group is not necessarily the most "influential" group. Standard DRO can fail by over-optimizing for an outlier that has no bearing on the test distribution. In contrast, TRO's use of centrality identifies groups that are better proxies for unseen data.

The data-driven topology often aligns with human intuition. For instance, in flood mapping, TRO identifies heavy-rainfall events as more central and influential than rare edge cases like dam collapses. Furthermore, by constraining the uncertainty set via graph structure, TRO provides tighter generalization bounds than traditional robust methods.

Limitations and Future Work

While effective, TRO faces challenges regarding computational cost, as data-driven topology learning requires calculating distances between all distributions. Currently, topology learning and optimization are separate steps; the authors suggest that developing an end-to-end differentiable pipeline is a vital next step. Additionally, the benefits of TRO are less pronounced in scenarios with very small distributional shifts, where group centralities tend to converge.

Summary updated

BibTeX
@inproceedings{qiao2023topology,
  author = {Fengchun Qiao and Xi Peng},
  title = {{Topology-aware Robust Optimization for Out-of-distribution Generalization}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=ylMq8MBnAp},
}
Figure