Abstract
We propose an improved estimator for the multi-task averaging problem, whose goal is the joint estimation of the means of multiple distributions using separate, independent data sets. The naive approach is to take the empirical mean of each data set individually, whereas the proposed method exploits similarities between tasks, without any related information being known in advance. First, for each data set, similar or neighboring means are determined from the data by multiple testing. Then each naive estimator is shrunk towards the local average of its neighbors. We prove theoretically that this approach provides a reduction in mean squared error. This improvement can be significant when the dimension of the input space is large; demonstrating a “blessing of dimensionality” phenomenon. An application of this approach is the estimation of multiple kernel mean embeddings, which plays an important role in many modern applications. The theoretical results are verified on artificial and real world data.
Introduction
Estimating the mean of a distribution is a fundamental statistical task. In multi-task scenarios, where one must estimate the means of numerous independent datasets (tasks) simultaneously, the standard approach is the naive estimator—calculating the empirical mean for each task separately. However, when tasks share underlying similarities (e.g., sensor data from similar satellites), the naive approach ignores valuable cross-task information.
This paper addresses the Multi-Task Averaging (MTA) problem by proposing a method to "borrow strength" across tasks. Unlike global shrinkage methods like the James-Stein estimator, which can fail when tasks are heterogeneous or clustered, the proposed approach identifies local similarities dynamically without prior knowledge of task relationships.
Methodology: Similarity Test Based (STB) Estimator
The authors introduce the Similarity Test Based (STB) approach, which utilizes a three-step process:
- Multiple Testing: Pairwise statistical tests determine if the true means of two tasks are "close."
- Neighborhood Selection: For each task, a neighborhood of similar tasks is identified based on those that pass the similarity test.
- Local Shrinkage: The task's mean estimate is shrunk toward the average of its identified neighbors.
A key theoretical contribution is the identification of the "blessing of dimensionality." In high-dimensional spaces, the detection radius for similarity tests shrinks faster than the estimation error of individual means. This allows for accurate neighbor identification even when initial estimates are noisy, effectively turning the "curse of dimensionality" into an asset. This methodology is particularly applicable to Kernel Mean Embeddings (KME), where distributions are mapped to high-dimensional Hilbert spaces.
Experimental Results
The STB estimator was evaluated against synthetic and real-world datasets, including Gaussian "bags" with varying structures and the AOD-MISR1 satellite dataset.
Synthetic Evaluation
In synthetic tests, the STB variants (STB-0, STB theory, STB weight) consistently outperformed the naive baseline and other state-of-the-art shrinkage methods like R-KMSE across scenarios involving imbalanced bag sizes and clustered distributions.
Real-World Application: AOD-MISR1
On the AOD-MISR1 satellite dataset, used for predicting aerosol optical depth across 800 bags, the STB methods demonstrated significant error reductions. Specifically, using an RBF kernel, the "STB weight" variant achieved a 22.73% reduction in estimation error compared to the naive approach.
Discussion and Conclusion
The STB method provides a robust, adaptive framework for multi-task averaging that automatically discovers task structures. By leveraging local rather than global shrinkage, it maintains accuracy in heterogeneous environments. While the method is robust, future work includes refining weighting schemes for highly imbalanced bag sizes and investigating the minimax optimality of the improvement factors. Overall, the STB approach demonstrates that high dimensionality can be leveraged to improve statistical estimation through more powerful similarity testing.
Summary updated