Robust Mean Estimation With Auxiliary Samples

Barron Han, Danil Akhtiamov, Reza Ghane, Babak HassibiarXiv 2025 · DOI · arXiv

This paper, recently published by researchers at Caltech, provides a rigorous theoretical foundation for a common practice in machine learning: using "cheap" auxiliary data to supplement "expensive" target data. Whether you are using synthetic data from a simulator to train a self-driving car or AI-generated images to augment a medical dataset, this research quantifies exactly how much that extra data is worth.

TL;DR

The authors establish the fundamental limits of mean estimation when combining samples from a target distribution with samples from a "close" auxiliary distribution. By using the Wasserstein-2 metric to bound the distance between these distributions, they derive the optimal robust estimator and prove that auxiliary data always improves performance, provided the uncertainty radius is small relative to the target's variance.

What Problem Does This Paper Address?

In many real-world scenarios, acquiring high-quality data from a "target" distribution (the one we actually care about) is costly or time-consuming. To compensate, practitioners often use "auxiliary" data—samples from a related but slightly different distribution (e.g., simulations, generative models, or data from a different geographic region).

The central challenge is distribution shift. If the auxiliary distribution is too different from the target, using it might actually degrade the performance of your estimator. Previous work, like Huber’s $\epsilon$-contamination model, focused on "poisoned" datasets where you don't know which samples are bad. This paper addresses a more modern "data augmentation" setting: we know exactly which samples are from the target and which are from the auxiliary source, but we need to know how to weight them optimally to minimize the worst-case error.

Methods

The authors frame the problem as a distributionally robust optimization (DRO) task.

  1. The Setup: They consider $n$ samples from a target distribution $\mathbb{P}$ and $N$ samples from an auxiliary distribution $\mathbb{Q}$.
  2. The Constraint: They assume $\mathbb{Q}$ lies within a Wasserstein-2 ($W_2$) "ball" of radius $\epsilon$ around $\mathbb{P}$. The $W_2$ distance is a natural choice because it is well-defined even for distributions that don't overlap and is widely used in optimal transport.
  3. The Estimator: They focus on linear estimators of the form $f = A\bar{X} + B\bar{Z}$, where $\bar{X}$ and $\bar{Z}$ are the empirical means of the target and auxiliary samples, respectively.
  4. Min-Max Optimization: The goal is to find the matrices $A$ and $B$ that minimize the Mean Square Error (MSE) under the worst-case auxiliary distribution $\mathbb{Q}$ allowed by the $W_2$ constraint.
  5. Reduction to Gaussians: A key technical lemma proves that, for this specific problem, the worst-case distributions are Gaussian. This allows the authors to simplify a complex functional optimization problem into a more tractable one involving only first and second moments.

Datasets

The paper is primarily theoretical and does not use large-scale real-world datasets. Instead, it validates its mathematical findings using:

  • Gaussian Location Models: Numerical simulations where $d=200$ (high-dimensional setting), $n=20$ (limited target data), and $N=1000$ (abundant auxiliary data). These simulations confirm that the derived "optimal" weights actually yield the lowest MSE compared to naive strategies.

Key Insights and Ideas

  • The Optimal Weighting Rule: The optimal estimator is a weighted average: $s\bar{X} + (1-s)\bar{Z}$. The weight $s$ assigned to the target data increases as the uncertainty ($\epsilon^2$) grows. If the auxiliary data is very "far" or the target data is very "clean," the estimator eventually ignores the auxiliary data ($s \to 1$).
  • Normalization Matters: To keep the error from blowing up as the variance of the distribution goes to infinity, the authors normalize the MSE by the norm of the target's covariance matrix ($\|\Sigma_X\|$). They show that the structure of the optimal estimator remains consistent whether you use the Frobenius norm, Trace, or Operator norm.
  • The "Aha" Moment on Utility: Auxiliary data is most useful when the squared $W_2$ radius ($\epsilon^2$) is small compared to the variance of the true distribution ($\delta^2$) divided by the number of true samples ($n$). Essentially, if your target data is scarce and noisy, even a slightly "off" auxiliary dataset is a massive help.

Key Results

  • Closed-Form Solutions: The authors provide explicit formulas for the optimal weighting constant $s$ and the resulting min-max MSE ($R^*$) for three different normalization schemes.
  • Performance Gain: They demonstrate that the robust estimator consistently outperforms two common baselines:
  • Target-only: Averaging only the $n$ samples.
  • Naive Average: Treating all $n+N$ samples as if they came from the same source.
  • Dimension Sensitivity: The results show how the problem dimension $d$ scales the benefit of auxiliary data, providing a roadmap for high-dimensional mean estimation.

Limitations and Future Work

  • Linearity Constraint: The paper assumes a linear estimator. While often optimal in Gaussian settings, non-linear estimators might perform better for more complex, non-convex distribution sets.
  • Mean Estimation Only: The work is limited to estimating the first moment (the mean). Extending this to robust covariance estimation or regression would be a significant next step.
  • Metric Choice: While $W_2$ is powerful, the authors suggest exploring other statistical distances like KL-divergence or other $W_p$ distances to see if the robust guarantees hold.
  • Practical Implementation: The current framework assumes we know the $W_2$ radius $\epsilon$ and the target covariance bound $\delta^2$. In practice, these must be estimated, which introduces its own set of challenges.

Summary updated

Abstract

In data-driven learning and inference tasks, the high cost of acquiring samples from the target distribution often limits performance. A common strategy to mitigate this challenge is to augment the limited target samples with data from a more accessible "auxiliary" distribution. This paper establishes fundamental limits of this approach by analyzing the improvement in the mean square error (MSE) when estimating the mean of the target distribution. Using the Wasserstein-2 metric to quantify the distance between distributions, we derive expressions for the worst-case MSE when samples are drawn (with labels) from both a target distribution and an auxiliary distribution within a specified Wasserstein-2 distance from the target distribution. We explicitly characterize the achievable MSE and the optimal estimator in terms of the problem dimension, the number of samples from the target and auxiliary distributions, the Wasserstein-2 distance, and the covariance of the target distribution. We note that utilizing samples from the auxiliary distribution effectively improves the MSE when the squared radius of the Wasserstein-2 uncertainty ball is small compared to the variance of the true distribution and the number of samples from the true distribution is limited. Numerical simulations in the Gaussian location model illustrate the theoretical findings.
BibTeX
@misc{han2025robust,
  author = {Barron Han and Danil Akhtiamov and Reza Ghane and Babak Hassibi},
  title = {{Robust Mean Estimation With Auxiliary Samples}},
  year = {2025},
  eprint = {2501.18095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.18095},
  howpublished = {arXiv},
}
Figure