High-Dimensional Differentially Private Quantile Regression: Distributed Estimation and Statistical Inference

Ziliang Shen, Caixing Wang, Shaoli Wang, Yibo Yan

TL;DR

This paper introduces a novel framework for performing differentially private quantile regression on high-dimensional, distributed datasets. It tackles the challenge of robustly estimating quantiles while preserving individual privacy, offering a robust solution for sensitive data analysis in distributed environments.

What Problem Does This Paper Address?

In today's data-rich world, sensitive personal information is often collected and stored across numerous distributed systems. This presents a dual challenge: how to analyze this data effectively while rigorously protecting individual privacy, and how to do so for quantile regression, a robust statistical method particularly useful for heavy-tailed data and outliers. Existing methods for differential privacy and distributed learning often operate separately, and few address the complexities of non-smooth quantile loss functions in high-dimensional settings, especially when combined with robust inference. This paper aims to bridge this gap by developing a unified framework that handles distributed data, high dimensionality, privacy guarantees, and robust quantile estimation and inference.

Methods

The core of the proposed method lies in a clever combination of techniques:

  • Newton-type Transformation: To handle the non-smooth quantile loss function, the authors adapt a Newton-type transformation. This transforms the quantile regression problem into a series of weighted least-squares problems, making it more amenable to optimization.
  • Distributed Gradient Aggregation: The distributed nature of the data is addressed by computing local gradients on each machine and then aggregating them centrally.
  • Noisy Hard Thresholding (NoisyHT): To enforce sparsity and ensure differential privacy simultaneously, the NoisyHT algorithm is applied iteratively. This mechanism adds calibrated noise and prunes less important coefficients, preserving privacy without destroying sparsity.
  • Differentially Private Debiasing and Inference: To correct for the bias introduced by privacy mechanisms and thresholding, a differentially private debiasing procedure is developed. This is followed by constructing differentially private confidence intervals and a communication-efficient differentially private bootstrap for simultaneous hypothesis testing.

Datasets

The authors evaluate their methods using extensive simulations with synthetic data generated under various noise distributions (normal, t-distribution, Cauchy) and error structures (homoscedastic and heteroscedastic). They also apply their framework to the 2022 Medical Expenditure Panel Survey (MEPS) Full Year Consolidated Data File (HC-243) for a real-world healthcare analytics application.

Key Insights and Ideas

  • Quantile Regression's Robustness Advantage: The paper highlights that quantile regression, unlike mean regression, remains stable and converges even with heavy-tailed noise distributions (like Cauchy) where linear regression fails. This underscores its value for real-world, often noisy, data.
  • The Inherent Cost of Privacy: A consistent finding across simulations is that differential privacy, while essential, introduces an additional error term. This "cost of privacy" leads to larger estimation errors compared to non-private methods, especially with tighter privacy budgets (smaller epsilon).
  • Privacy-Utility Trade-off: The study clearly demonstrates the trade-off between privacy and statistical accuracy. Smaller privacy budgets (lower epsilon) lead to more robust privacy guarantees but also higher estimation errors.
  • Distributed Bootstrap for Simultaneous Inference: The development of a differentially private multiplier bootstrap procedure is a significant contribution, enabling valid simultaneous confidence intervals and hypothesis tests in a distributed and privacy-preserving manner.

Key Results

  • Near-Oracle Rates with Privacy: The proposed differentially private estimation method achieves convergence rates that are close to the oracle (non-private) rates, up to logarithmic factors, even in high-dimensional sparse settings.
  • Robustness to Heavy Tails: The quantile regression estimators demonstrate stability and convergence under heavy-tailed noise distributions, outperforming linear regression baselines which fail in such scenarios.
  • Valid Inference: The debiased estimators and bootstrap procedures provide valid confidence intervals and hypothesis tests, even under differential privacy constraints.
  • Real-World Applicability: The application to the MEPS dataset shows that the method can identify interpretable healthcare utilization factors associated with medical expenditure, demonstrating its practical utility.

Limitations and Future Work

  • Initialization Sensitivity: The paper acknowledges that the performance of Newton-type methods can be sensitive to initialization and bandwidth choices. While the NoisyHT step helps stabilize updates, accurate initialization remains important.
  • Trusted Central Machine Assumption: The current framework assumes a trusted central machine for gradient aggregation and privacy perturbation. Future work could explore fully decentralized or more robust distributed settings.
  • Non-i.i.d. and Heterogeneous Data: Extending the theoretical guarantees to non-i.i.d. data and more complex client heterogeneity is a significant future direction.
  • Joint Inference: Developing a full joint asymptotic normality theory for debiased estimators over growing index sets is identified as a challenging but important area for future research.
  • Advanced Privacy Mechanisms: Exploring other privacy mechanisms like local or Gaussian differential privacy, and advanced optimization techniques like DP-ADMM, could further improve the privacy-utility trade-off.

Summary updated

Abstract

With the development of big data and machine learning, privacy concerns have become increasingly critical, especially when handling heterogeneous datasets containing sensitive personal information. Differential privacy provides a rigorous framework for safeguarding individual privacy while enabling meaningful statistical analysis. In this paper, we propose a differentially private quantile regression method for high-dimensional data in a distributed setting. Quantile regression is a powerful and robust tool for modeling the relationships between the covariates and responses in the presence of outliers or heavy-tailed distributions. To address the computational challenges due to the non-smoothness of the quantile loss function, we introduce a Newton-type transformation that reformulates the quantile regression task into an ordinary least squares problem. Building on this, we develop a differentially private estimation algorithm with iterative updates, ensuring both near-optimal statistical accuracy and formal privacy guarantees. For inference, we further propose a differentially private debiased estimator, which enables valid confidence interval construction and hypothesis testing. Additionally, we propose a communication-efficient and differentially private bootstrap for simultaneous hypothesis testing in high-dimensional quantile regression, suitable for distributed settings with both small and abundant local data. Extensive simulations demonstrate the robustness and effectiveness of our methods in practical scenarios.
BibTeX
@misc{shen2025high,
  author = {Ziliang Shen and Caixing Wang and Shaoli Wang and Yibo Yan},
  title = {{High-Dimensional Differentially Private Quantile Regression: Distributed Estimation and Statistical Inference}},
  year = {2025},
  eprint = {2508.05212},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.05212},
}
Figure