Monotonic Gaussian Process Flow

Ivan Ustyuzhaninov, Ieva Kazlauskaite, Carl Henrik Ek, Neill D. F. Campbell

Abstract

Gaussian Processes (GPs) are a powerful tool for Bayesian regression, yet enforcing global monotonicity remains a significant challenge. We present Monotonic Gaussian Process Flows, a novel framework that guarantees monotonicity by modeling functions as the solutions to Stochastic Differential Equations (SDEs). By leveraging the property that paths in a valid flow field cannot cross, our method ensures monotonicity by construction. We demonstrate that this approach provides superior uncertainty calibration and performance, particularly in low-data regimes, compared to existing constrained GP methods.

Introduction

In many scientific and engineering domains, prior knowledge dictates that a relationship must be monotonic. Examples include growth curves, dose-response relationships, and temporal alignment in signal processing. While standard GPs offer flexibility and principled uncertainty quantification, they do not inherently respect monotonicity constraints. Existing solutions often rely on soft constraints (penalizing negative derivatives at specific points) or rigid parametric forms like constrained splines. Soft constraints fail to provide global guarantees and can distort uncertainty intervals, while parametric models lack the expressiveness of non-parametric approaches.

Methodology: Monotonicity via Flow Uniqueness

The core innovation of this work is the use of "flows" to define monotonic functions. In fluid dynamics, if a flow is governed by a well-behaved differential equation, the relative order of particles is preserved—they can never cross. We translate this principle into a machine learning model by defining a function $f(x)$ as the terminal position of a particle starting at $x$ after evolving through a Stochastic Differential Equation (SDE) for a fixed time $T$.

The SDE is defined by a drift and diffusion process where both are modeled using a Sparse Gaussian Process. Because the solutions to such SDEs are unique and continuous, the mapping from the initial state $x$ to the final state $f(x)$ is guaranteed to be monotonic. Inference is performed using variational methods, allowing the model to scale to larger datasets while maintaining a fully Bayesian treatment of uncertainty.

Key Insights and Ideas

A major advantage of this framework is that monotonicity is a structural property of the model rather than an optimization constraint. This leads to more "natural" uncertainty estimates. In regions far from data, the requirement of monotonicity restricts the space of valid functions, causing uncertainty bands to contract in a way that standard GPs cannot replicate.

Furthermore, these monotonic flows are highly effective as "warping" layers in hierarchical models. For instance, in time-series alignment, one can warp the time axis of multiple sequences to a common reference frame. Since time is inherently monotonic, our flow model provides a principled way to ensure that the warping never "reverses" time.

Conceptual diagram of hierarchical warping where monotonic flows align unknown latent functions by finding optimal warps.

Key Results

The model was evaluated against a "Transformed GP" baseline across six benchmark functions (linear, exponential, logistic, step, etc.) in both low-data ($N=15$) and high-data ($N=100$) regimes.

In the low-data regime, the Monotonic Flow significantly outperformed the baseline. For the "step" function, the Flow achieved an RMSE of 64.5, a substantial improvement over the baseline's 101.9. As shown in the figure below, the Flow model (orange lines) consistently follows the true function (red dashed line) more closely than the baseline (blue lines), especially when data is sparse.

Comparison of Monotonic GP Flows against Transformed GP baselines across linear, exponential, and logistic functions with varying data density.

In the high-data regime ($N=100$), the model remained highly competitive, achieving state-of-the-art results on half of the benchmark functions. Crucially, when extrapolating beyond the observed data range, the Flow model produced samples that remained strictly monotonic and exhibited realistic uncertainty growth, whereas other methods often failed to maintain the constraint in the tails.

Limitations and Future Work

While providing strong theoretical guarantees, the method is more computationally intensive than standard GPs due to the need for numerical SDE solvers. The complexity scales as $O(N_{steps}(NM^2 + N^2))$, where $M$ is the number of inducing points. Additionally, the monotonicity guarantee in practice depends on the step size of the numerical solver being sufficiently small. Future research will focus on extending this framework to multi-dimensional inputs and optimizing the SDE integration process to reduce computational overhead.

Conclusion

Monotonic Gaussian Process Flows offer a principled, non-parametric solution to the long-standing problem of enforcing monotonicity in Bayesian regression. By framing the problem through the lens of differential equation flows, we achieve a model that is monotonic by construction, providing robust performance and well-calibrated uncertainty across a variety of challenging tasks.

Summary updated

BibTeX
@inproceedings{ustyuzhaninov2020monotonic,
  author = {Ivan Ustyuzhaninov and Ieva Kazlauskaite and Carl Henrik Ek and Neill D. F. Campbell},
  title = {{Monotonic Gaussian Process Flow}},
  booktitle = {Apollo},
  year = {2020},
  doi = {10.17863/cam.52012},
}
Figure