Abstract
Probabilistic modeling is essential for uncertainty quantification in high-stakes decision-making. Traditional methods often rely on rigid parametric assumptions or unstable non-parametric splines. This paper introduces Neural Spline Search (NSS), a framework that automates the discovery of expressive quantile functions by combining basis splines through symbolic operators. By treating the distribution shape as a searchable architecture, NSS provides a flexible, distribution-agnostic approach to regression and forecasting. Experimental results across UCI benchmarks and the M5 forecasting competition demonstrate that NSS achieves state-of-the-art performance and superior calibration.
Introduction
In domains such as medical diagnostics, financial risk assessment, and retail inventory management, point estimates—single "best guess" predictions—are often insufficient. Decision-makers require a comprehensive understanding of uncertainty, typically expressed through quantiles that define prediction intervals. However, modeling these intervals accurately is challenging.
Standard probabilistic models frequently assume that data follows a specific parametric distribution, such as a Gaussian. Real-world data, however, is rarely so well-behaved; it often exhibits skewness, multi-modality, and heteroscedasticity (variance that changes with input features). When the assumed distribution fails to match the underlying data generation process, prediction intervals become unreliable. While non-parametric methods like splines offer more flexibility, individual splines can become numerically unstable when forced to model highly complex distributions.
The Neural Spline Search (NSS) Framework
Neural Spline Search (NSS) addresses these challenges by moving away from fixed assumptions. Instead, it treats the quantile function—the inverse of the Cumulative Distribution Function (CDF)—as a composition of learnable, modular components. The framework "searches" for an optimal architecture by combining basis splines using symbolic operators.
Basis Splines
NSS utilizes two primary types of monotonic splines as building blocks:
- C-splines: These place knots within the CDF space, effectively mapping quantile levels to specific values.
- P-splines: These place knots within the Probability Density Function (PDF) space, allowing the model to represent the density of the distribution directly.
Symbolic Operators
The core innovation of NSS lies in how these basis splines are composed. The authors introduce two primary operators:
- NSS-sum: This operator computes a weighted sum of different splines. This approach balances the robustness of simpler splines with the high expressivity of more complex ones, preventing the instability often seen in high-degree individual splines.
- NSS-chain: Inspired by the architecture of Normalizing Flows, this operator passes the output of one spline as the input to the next. The framework supports α-chaining (transforming the quantile level) and X-chaining (transforming the conditional features).
Unlike traditional Normalizing Flows, which require mathematically invertible "bijectors" to calculate likelihoods, the NSS-chain only requires a forward pass to generate quantiles. This "forward-only" nature significantly reduces computational complexity and expands the range of usable transformations.
Training and Optimization
The NSS model is trained by minimizing the Continuous Ranked Probability Score (CRPS), a proper scoring rule that rewards both accuracy and sharp probabilistic calibration. Because the analytical solution for CRPS is often intractable for complex spline compositions, the authors employ Monte Carlo estimation, sampling multiple quantile levels during each training iteration to approximate the loss.
Experimental Evaluation
The researchers evaluated NSS across three distinct environments:
- Synthetic Data: A 1D dataset featuring a sinusoidal mean and data-dependent variance, used to visualize the model's ability to capture varying noise levels.
- UCI Machine Learning Repository: Six tabular regression benchmarks (including Boston Housing, Concrete, and Protein) were used to test general regression performance.
- M5 Forecasting Competition: A large-scale retail dataset from Walmart was used to evaluate performance on complex, high-dimensional time-series data.
Key Results
- Tabular Regression: NSS-sum outperformed all baseline models—including Gaussian processes and specialized quantile regressors like RQspline—on five out of six UCI datasets. The model demonstrated a significant improvement in Mean Absolute Error (MAE), ranging from 1.1% to 17.7%.
- Quantile Accuracy: Using "Pinball Loss" as a metric, NSS consistently achieved the lowest error rates, particularly on the complex "Protein" and "Concrete" datasets.
- Large-Scale Forecasting: In the M5 competition, NSS-chain achieved superior results in terms of Weighted Average Percentage Error (WAPE) and Root Mean Square Error (RMSE), highlighting its efficacy in industrial-scale applications.
- Calibration: Calibration analysis confirmed that NSS models are well-calibrated; predicted 90% confidence intervals accurately contained the true values approximately 90% of the time.
Discussion and Insights
The success of NSS stems from several key architectural choices. By prioritizing composition over complexity, the framework gains expressivity through the combination of simple, stable splines rather than increasing the depth or knot-density of a single function. This makes the model distribution-agnostic, allowing it to adapt to any data shape without prior assumptions.
Furthermore, the authors noted task-specific strengths: NSS-sum is particularly effective for static regression tasks where balancing different distribution shapes is key, while NSS-chain excels in time-series forecasting, likely due to its ability to better capture sequential dependencies.
Limitations and Future Work
While NSS represents a significant step forward, the authors identified areas for further exploration. The current search space was limited to a depth of two splines; deeper architectures or additional symbolic operators (such as multiplication or trigonometric functions) could yield even higher performance. Additionally, creating hybrid models that incorporate classical parametric distributions (like the Gaussian) as a "basis" within the search space could combine the inherent stability of classical statistics with the flexibility of neural splines.
Summary updated