Scalable Optimization for Non-Smooth Quantile Regression

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 226 papers29 conceptsfrozen 2026-10-04

Standard linear regression is built on a simple premise: minimize the sum of squared errors to find the conditional mean. While mathematically convenient, the mean is a fragile statistic. It is easily pulled away by outliers and tells us nothing about the "tails" of a distribution—the extreme events that often matter most in finance, medicine, and network reliability.

To see the whole picture, researchers turn to Quantile Regression (QR). Instead of the average, QR allows us to model any slice of the distribution, such as the 90th percentile of network latency or the 5th percentile of infant birth weights. However, this robustness comes at a steep computational price.

The Problem of the Kink

The core challenge of quantile regression lies in its loss function, often called the "check function" or "pinball loss." Unlike the smooth, U-shaped parabola of least squares, the check function is a V-shape with a sharp "kink" at the origin.

Mathematically, this function is non-differentiable at zero. This single point of non-smoothness breaks the most powerful tools in the optimization shed. You cannot take a standard derivative at the kink, which means classical gradient descent and Newton-type methods—which rely on smooth first and second derivatives—cannot be directly applied. As noted by Matthew Pietrosanu and Linglong Kong in their work on the cqrReg package, this lack of smoothness leads to significant computational overhead, especially as datasets grow in dimensionality.

Smoothing the Path

To regain the speed of gradient-based optimization, one prominent research direction involves "smoothing" the kink. By approximating the non-differentiable check function with a smooth surrogate, researchers can use efficient first-order methods.

  1. Convolution Smoothing: The conquer framework, developed by Xiaoou Pan and colleagues, uses a convolution-type smoothing approach. This transforms the problem into one that is twice-differentiable, allowing for fast, scalable inference even with tens of millions of observations.
  2. Finite Smoothing: In the context of non-linear models, the fastkqr algorithm by Qian Tang and Yuwen Gu applies smoothing to Kernel Quantile Regression. This allows them to fit multiple, non-crossing quantile curves simultaneously—a task that is traditionally a combinatorial nightmare.
  3. Functional Gradient Descent: Wanzhou Ye proposed the Smooth Quantile Boost Classification (SQBC) algorithm, which applies a smooth approximation to the check function to enable direct boosting, making binary classification more robust to label noise.

Expectiles: The Differentiable Cousin

If the non-smoothness of quantiles is too high a hurdle, another option is to change the target statistic entirely. Expectiles are to the mean what quantiles are to the median. They are defined by minimizing an asymmetric least squares loss, which is continuous and differentiable everywhere.

Di Niu and Linglong Kong demonstrate in their work on Expectile Matrix Factorization that expectiles can capture the "shape" of skewed data (like network latencies) while remaining computationally efficient. Because the loss is differentiable, it scales to large-scale matrix problems where standard quantile regression would stall. Similarly, Ingo Steinwart has shown that kernel-based expectile regression can achieve optimal learning rates using adaptations of the Sequential Minimal Optimization (SMO) algorithm, bridging the gap between robust statistics and Support Vector Machines.

Quantiles at Scale: Distributed and High-Dimensional Learning

Modern data is often too large for one machine or is siloed across different locations. The non-smoothness of the quantile loss makes "averaging" local results difficult, as the standard bias-variance decompositions used in least squares don't apply.

  • Divide-and-Conquer: Heng Lian proved that a distributed nonparametric quantile regression strategy—averaging local estimates from multiple machines—can achieve centralized learning rates, provided the number of machines is not excessively large.
  • Double Smoothing: For high-dimensional settings where we must also select relevant features, Caixing Wang and Ziliang Shen introduced DHSQR. This method uses a "double-smoothing" technique to turn the non-smooth problem into a sequence of smooth least-squares problems, allowing for perfect feature recovery in distributed environments.
  • One-Shot Ensembles: In federated learning, where communication is expensive, Guang Yang and Xingcai Zhou developed one-shot weighted ensembles. This approach allows agents to collaborate on a quantile model with only a single round of communication, even when they observe different sets of features.

Beyond Points: Functional Data

The frontier of this research extends to Functional Data Analysis, where the "data point" is actually a curve, such as a brain scan or a growth trajectory over time. Nan Zhang and Jianhua Z. Huang have pushed quantile regression into two-dimensional domains, reformulating the non-smooth estimation as a quadratic programming problem. This allows researchers to see how the relationship between variables (like brain structure and cognition) changes not just on average, but at the edges of the population.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 27 papers
  2. Version 1 · 2026-10-04 · 26 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_non_smooth_loss_optimization_86136ebd,
  title = {non-smooth loss optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-smooth-loss-optimization/rdr_86136ebdd12b439a99ac50dac18f6652},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scalable_optimization_for_non_smooth_qua_86136ebd_v1,
  title = {Scalable Optimization for Non-Smooth Quantile Regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scalable-optimization-for-non-smooth-quantile-regression/rdr_86136ebdd12b439a99ac50dac18f6652/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}