Distributionally Robust Set Representation Learning Under Inference-Time Element Corruption

Yankai Chen, Hanrong Zhang, Bowei He, Philip S. Yu, Xue Liu

TL;DR

This paper introduces SW-DRSO, a novel framework for robustly learning set representations that can withstand element-level corruptions (like outliers or missing data) during inference. By leveraging Sliced-Wasserstein distance and a barycentric adversary, SW-DRSO effectively trains models to be resilient to these real-world data imperfections without sacrificing performance on clean data.

What Problem Does This Paper Address?

Standard Set Representation Learning (SRL) methods are great at encoding unordered collections of data (like point clouds or sets of features) into meaningful vectors. However, these methods often perform poorly when deployed in real-world scenarios where the input data might be corrupted at inference time. This "inference-time element corruption" can manifest as outliers, missing elements, or noisy data points, which can significantly distort the learned set representation and lead to incorrect predictions. Existing SRL techniques typically assume clean, well-formed input data, leaving a critical gap in their practical applicability.

Methods

The core of SW-DRSO lies in its application of Distributionally Robust Optimization (DRO) to SRL. Instead of just minimizing errors on the training data (Empirical Risk Minimization - ERM), SW-DRSO aims to minimize the worst-case expected loss over a family of plausible corrupted data distributions.

Here's a breakdown of their key techniques:

  • Sliced-Wasserstein (SW) Distance for Ambiguity Region: They define the "ambiguity region" – the set of plausible corruptions – using the Sliced-Wasserstein distance. This metric is well-suited for sets and allows for flexible modeling of corrupted sets without needing to enumerate all possibilities.
  • Wasserstein-Aware Set Encoder: They utilize a set encoder that is specifically designed to respect the geometric properties of the SW metric, ensuring that the learned representations are aligned with the corruption modeling.
  • Barycentric Adversary: A major challenge in DRO is efficiently finding the "worst-case" corrupted input. SW-DRSO introduces a "barycentric adversary" that approximates this intractable search. It synthesizes adversarial examples by taking convex combinations (barycenters) of neighboring set embeddings. This turns a discrete, combinatorial optimization problem into a continuous, differentiable optimization over a simplex, making it computationally tractable.

Datasets

The authors evaluated SW-DRSO on four diverse tasks using established datasets:

  • Task I: Similar Set Ranking: Friendster and LIVEJ (social network datasets).
  • Task II: Point Cloud Classification: ModelNet40.
  • Task III: Topic Set Expansion: LDA-1k, LDA-3k, and LDA-5k (datasets for topic modeling).
  • Task IV: Patch-set Visual Recognition: NWPU-RESISC45 (remote sensing images).

They employed a standardized data partitioning protocol, training on clean data and evaluating on clean, mild, and severe corruption levels to simulate real-world deployment conditions.

Key Insights and Ideas

  • The "Curated Data" vs. "Real-World Data" Gap: The paper highlights a significant practical limitation of current SRL methods: their reliance on clean, curated data. This work directly addresses the need for robustness in the face of common inference-time data degradations.
  • Geometric Alignment is Key: The success of SW-DRSO seems to stem from the geometric alignment between the Sliced-Wasserstein distance used for defining corruptions and the set encoder. This suggests that leveraging the underlying geometry of the data is crucial for robust set representation learning.
  • Barycentric Adversary as an Efficient Surrogate: The barycentric adversary is a clever solution to the computational bottleneck of DRO. By approximating the worst-case search with a differentiable optimization over a simplex, they achieve robustness without prohibitive training costs.
  • Robustness Without Sacrificing Clean Performance: A crucial finding is that SW-DRSO not only enhances robustness against corruption but also maintains high performance on clean data, demonstrating a favorable trade-off.

Key Results

  • Consistent Performance Gains: SW-DRSO consistently outperformed various state-of-the-art baselines across all four tasks and different corruption levels.
  • Reduced Performance Degradation: The most significant finding is the reduced performance gap between clean and severely corrupted data for SW-DRSO compared to other methods. For instance, in Task I (Similar Set Ranking), the shaded region representing the performance range from clean to severe splits was notably narrower for SW-DRSO.
  • Statistical Significance: The performance improvements of SW-DRSO were statistically significant in most comparisons.
  • Computational Efficiency: While achieving better robustness, SW-DRSO also demonstrated competitive or improved training efficiency compared to some other DRO variants. For example, their SW approximation was 27% faster than a standard Wasserstein approach.

Limitations and Future Work

  • Computational Cost of SW Encoder: While the barycentric adversary makes the DRO part efficient, the SW-aware encoder itself can still be computationally intensive, especially with a large number of projection directions (R).
  • Generalizability to Other DRO Frameworks: The paper notes that SW-DRSO works best when the robust optimization objective is combined with a geometry-aligned encoder (like SW-aware or FSW/FSPool). This suggests that simply plugging SW-DRSO into any encoder might not yield optimal results, implying a degree of encoder-DR framework compatibility is beneficial.
  • Future Directions: The authors suggest extending their robust set optimization techniques to more complex domains like Large Language Models (LLMs) and agent-centric settings, where robustness to distributional shifts and corruptions is increasingly critical.

Summary updated

Abstract

Standard Set Representation Learning methods typically excel on curated data but often overlook the challenge of Inference-time Element Corruption. This refers to scenarios where deployed models encounter element-level degradations, such as outliers or missing components, that may distort set representation and degrade performance. We propose SW-DRSO, a distributionally robust optimization framework tailored for sets. Rather than minimizing loss solely on observed training data, SW-DRSO optimizes a tractable surrogate of the worst-case expected loss over a family of plausible inference-time variations. We introduce a barycentric adversary that approximates the intractable search over corrupted sets by a differentiable training-time optimization over simplex weights. Extensive experiments across four tasks demonstrate that SW-DRSO effectively enhances robustness against corruption while maintaining high overall performance.
BibTeX
@inproceedings{chen2026distributionally,
  author = {Yankai Chen and Hanrong Zhang and Bowei He and Philip S. Yu and Xue Liu},
  title = {{Distributionally Robust Set Representation Learning Under Inference-Time Element Corruption}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=lZqQN9Ji21},
}
Figure