Gaussian processes with monotonicity information

Jaakko Riihimäki, Aki VehtariAISTATS 2010

Abstract

Gaussian Processes (GPs) are powerful non-parametric models for regression and classification, but they often lack the ability to incorporate known physical or logical constraints, such as monotonicity. This paper presents a framework for enforcing monotonicity in GPs by treating it as "virtual" derivative observations. Using Expectation Propagation (EP) to handle the resulting non-Gaussian constraints, the authors demonstrate that incorporating monotonicity information leads to more robust predictions, especially in data-sparse regions, and helps guide hyperparameter optimization by pruning the marginal likelihood surface.

Introduction

In many real-world scenarios, domain knowledge dictates that a function should be monotonic—for example, the risk of a disease usually increases with age, or a physical response increases with stimulus. Standard flexible models like Gaussian Processes are purely data-driven; if the training data is noisy or sparse, the GP might "wiggle" or predict a downward trend where logic dictates an upward one. This paper provides a principled, computationally efficient way to bake this monotonicity information directly into the GP framework.

Methodology: Virtual Derivative Observations

The authors leverage a key property of Gaussian Processes: the derivative of a GP is itself a GP. This allows them to model the relationship between function values and their partial derivatives analytically.

  1. Virtual Derivative Observations: Instead of enforcing monotonicity everywhere (which is computationally impossible), they pick a finite set of "operating points." At these points, they introduce a "virtual observation" that the derivative must be non-negative.
  2. Probit Likelihood: To enforce the constraint, they use a probit function as a likelihood for the derivative. This acts like a "soft" step function: it gives high probability to positive derivatives and very low probability to negative ones. A parameter $\nu$ controls the "strictness" of this constraint.
  3. Expectation Propagation (EP): Because the probit likelihood is non-Gaussian, the posterior distribution becomes analytically intractable. The authors use the EP algorithm to approximate the posterior with a Gaussian, allowing them to use standard GP prediction machinery.
GP regression with monotonicity constraints. Top row: function values. Bottom row: derivatives. (a, d) Standard GP. (b, e) Monotonic GP with virtual points at data locations. (c, f) Monotonic GP with grid-based virtual points.

Key Insights and Ideas

Pruning the Likelihood Surface

One of the most striking insights is how monotonicity constraints affect the marginal likelihood. In standard GPs, the likelihood surface might have multiple "modes" (local optima)—some representing smooth functions and others representing wiggly, overfitted ones. Adding monotonicity information effectively "kills off" the overfitted modes, guiding the optimizer toward more physically sensible hyperparameters.

Smoothness vs. Monotonicity

The paper notes a tension between the GP's smoothness prior (controlled by the covariance function) and the monotonicity constraint. For a sharp "step" function, a monotonic GP might actually perform worse in RMSE because the smoothness prior prevents it from jumping sharply, while the monotonicity constraint prevents it from "wiggling" to fit the corners.

Value in Data Scarcity

The benefits of this method are most pronounced where data is scarce. In these regions, the monotonicity constraint acts as a powerful regularizer that prevents the model from making nonsensical predictions due to outliers.

Experimental Results

Healthcare Application: Institutionalization Risk

The paper evaluates the method on a real-world healthcare dataset from Vantaa, Finland. The goal was to predict the risk of a patient being institutionalized within three months, with the prior knowledge that risk should increase with age.

Comparison of predicted institutionalization risk profiles. (a) Standard GP without monotonicity information shows non-physical dips in risk. (b) Monotonic GP provides more realistic, steadily increasing risk profiles.

In the institutionalization study, the monotonic GP provided significantly better predictive performance (measured by ROC curves) for the oldest age group, where data was most sparse. Qualitatively, the monotonic GP produced "risk profiles" that healthcare professionals would find more credible—showing a steady increase in risk with age rather than the slight, non-physical dips seen in the standard model.

Conclusion

By treating monotonicity as virtual derivative observations, the proposed framework allows for the seamless integration of domain knowledge into Gaussian Process models. While computational complexity remains a challenge for high-dimensional data, the method significantly improves the robustness and qualitative realism of predictions in data-sparse scenarios.

Summary updated

BibTeX
@article{riihimaki2010gaussian,
  author = {Jaakko Riihimäki and Aki Vehtari},
  title = {{Gaussian processes with monotonicity information}},
  journal = {AISTATS},
  year = {2010},
}
Figure