Adaptive Data Fusion for Multi-task Non-smooth Optimization

Henry Lam, Kaizheng Wang, Yuhang Wu, Yichen ZhangarXiv 2022 · OpenReview

Abstract

We study the problem of multi-task non-smooth optimization that arises ubiquitously in statistical learning, decision-making and risk management. We develop a data fusion approach that adaptively leverages commonalities among a large number of objectives to improve sample efficiency while tackling their unknown heterogeneities. We provide sharp statistical guarantees for our approach. Numerical experiments on both synthetic and real data demonstrate significant advantages of our approach over benchmarks.

Introduction

In statistical learning and decision-making, practitioners often face a fundamental trade-off between local precision and global information. Training a model on a single task with limited data (Single-Task Learning) often leads to poor generalization. Conversely, "blindly" pooling data from many tasks (Data Pooling) risks "negative transfer," where the inherent differences between tasks degrade the performance of the model. This paper introduces an adaptive data fusion framework designed to find the optimal balance, specifically for non-smooth optimization problems.

The Challenge of Non-Smoothness

While multi-task learning (MTL) is well-studied for smooth loss functions, many real-world applications are inherently non-smooth. A classic example is the Newsvendor Problem in supply chain management, where the cost of overstocking differs from the cost of understocking, resulting in a "check loss" function with a distinct kink. Other examples include Quantile Regression and Support Vector Machines (SVMs).

Non-smooth problems present two primary challenges:

  1. Data Scarcity: Individual tasks (e.g., predicting sales for a single small retail outlet) often lack sufficient data to build robust models independently.
  2. Unknown Heterogeneity: While tasks are often related, the exact degree of similarity is unknown. Pooling data from a high-volume urban store with a low-volume rural store without accounting for these differences leads to sub-optimal decision-making.

Methodology: Adaptive Information Sharing

The researchers propose an augmented objective function that balances individual task performance with a penalty that encourages tasks to converge toward a common center ($\beta$):

$$F_{w,\lambda}(\Theta, \beta) = \sum_{j=1}^m w_j [f_j(\theta_j) + \lambda_j ||\theta_j - \beta||_2]$$

In this formulation, $f_j(\theta_j)$ represents the loss for an individual task, while the $\ell_2$ penalty $|\theta_j - \beta|_2$ acts as a "tuning knob." The critical innovation lies in the use of the $\ell_2$ penalty rather than a squared $\ell_2$ penalty. This creates a mathematical "cusp" at zero, which allows the model to perform exact data pooling when tasks are sufficiently similar, while remaining robust enough to ignore "outlier" tasks that do not fit the general pattern.

Theoretical and Empirical Validation

The authors provide the first sharp statistical guarantees for this approach within the context of non-smooth losses. They prove that their estimator achieves minimax optimality—the best possible error rate mathematically allowed—up to logarithmic factors. This ensures that the model automatically adapts to task similarity without requiring prior knowledge of the relationships between tasks.

Experimental Setup

The framework was tested on two primary datasets:

  • Synthetic Data: A multi-task quantile regression setup with 50 tasks, testing the model's resilience to varying levels of task similarity and outliers.
  • Real-World Bakery Data: A dataset comprising 1,215 days of sales across 35 stores in a German bakery chain. This represents a covariate-assisted newsvendor problem where stores must optimize daily stock based on weather, holidays, and promotions.

Key Findings

  • Efficiency in Scarcity: In the bakery experiment, the adaptive fusion method significantly outperformed Single-Task Learning when only 1–8 months of training data were available.
  • Avoiding the Pooling Trap: While simple Data Pooling performed well with very little data, its error rate eventually plateaued because it could not account for store-specific nuances. The adaptive method continued to improve as more data arrived, eventually matching the personalization of STL but with much higher initial efficiency.
  • Robustness: The theory and experiments confirmed that even if a fraction of tasks are outliers, the method improves performance on the related tasks without being negatively influenced by the divergent ones.

Conclusion and Future Work

This research provides a robust mathematical bridge between local personalization and global data sharing. By successfully applying adaptive fusion to non-smooth problems, the authors open new avenues for more efficient risk management and statistical learning. Future research directions include improving computational scalability for very large task sets, developing uncertainty quantification (confidence intervals) for the estimates, and integrating privacy-preserving mechanisms like Federated Learning for cross-organizational data sharing.

Summary updated

Abstract

We study the problem of multi-task non-smooth optimization that arises ubiquitously in statistical learning, decision-making and risk management. We develop a data fusion approach that adaptively leverages commonalities among a large number of objectives to improve sample efficiency while tackling their unknown heterogeneities. We provide sharp statistical guarantees for our approach. Numerical experiments on both synthetic and real data demonstrate significant advantages of our approach over benchmarks.
BibTeX
@misc{lam2022adaptive,
  author = {Henry Lam and Kaizheng Wang and Yuhang Wu and Yichen Zhang},
  title = {{Adaptive Data Fusion for Multi-task Non-smooth Optimization}},
  year = {2022},
  eprint = {2210.12334},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.12334},
}
Figure