Indian Buffet process for model selection in convolved multiple-output\n Gaussian processes

Cristian Guarnizo, Mauricio A. ÁlvarezarXiv 2015 · DOI · arXiv

Abstract

Multi-output Gaussian processes (MOGPs) have emerged as a powerful framework for modeling multiple related variables by exploiting their underlying correlations. A critical challenge in MOGP design is determining the optimal number of latent processes required to represent the data accurately without overfitting. This paper proposes the use of the Indian Buffet Process (IBP) as a non-parametric Bayesian prior to perform automatic model selection over the number of latent Gaussian processes. By integrating IBP with process convolutions and variational inference, the model learns sparse, interpretable relationships between latent forces and observed outputs. The effectiveness of this approach is demonstrated through synthetic experiments, human motion capture data, and gene expression analysis, showing its particular utility in the context of Latent Force Models (LFMs).

Introduction

Gaussian Processes (GPs) are widely recognized for their flexibility in Bayesian regression. However, extending them to multi-output scenarios requires carefully designed covariance functions that capture inter-output dependencies while maintaining positive definiteness. Common approaches, such as the Linear Model of Coregionalization (LMC) and process convolutions, rely on a set of latent Gaussian processes to drive the observed outputs.

A recurring difficulty in these models is "model selection": deciding how many latent processes are necessary. An insufficient number of processes fails to capture complex correlations, while an excessive number leads to computational inefficiency and potential overfitting. This paper addresses this by employing the Indian Buffet Process (IBP), allowing the model to automatically determine the number of active latent forces from the data.

Methodology

1. Process Convolutions (PC)

Rather than defining a correlation matrix directly, the authors utilize process convolutions. In this framework, each output is modeled as the convolution of a latent white-noise process with a specific kernel (smoothing function). This approach is particularly powerful because it allows for the incorporation of physical constraints, such as those derived from differential equations, directly into the kernel structure.

2. The Indian Buffet Process (IBP) Prior

The IBP is a non-parametric prior used to define a distribution over binary matrices with a finite number of rows (outputs) and an infinite number of columns (latent features). In this model, the IBP acts as a selector for the "sensitivity" matrix. It determines which latent processes influence which outputs. Because the IBP naturally encourages sparsity, it acts as an automatic Occam’s Razor, utilizing only the minimum number of latent processes required to explain the observed variance.

3. Variational Inference

To make the infinite-dimensional nature of the IBP computationally tractable, the authors derive a variational lower bound. This approach approximates the posterior distribution of the latent variables and the IBP matrix, offering a more efficient alternative to traditional Markov Chain Monte Carlo (MCMC) sampling methods.

Latent Force Models and Physical Interpretability

The proposed model is especially relevant for Latent Force Models (LFMs). In LFMs, latent processes represent physical quantities—such as protein concentrations in biological systems or mechanical forces in physical ones. By using kernels derived from the Green’s functions of first- or second-order Ordinary Differential Equations (ODEs), the model ensures that the learned latent signals correspond to physically meaningful drivers.

Experimental Results

Synthetic Data

In controlled experiments with synthetic datasets, the model successfully recovered the ground-truth structure of the connectivity matrix $\mathbf{Z}$. Even when initialized with a maximum number of latent processes ($Q$) far exceeding the true count, the IBP prior effectively pruned the unnecessary processes, setting their influence to zero.

Human Motion Capture

Using data from the CMU Graphics Lab (Subject 35, walking), the model was tasked with representing 20 joint angle channels. The IBP-based MOGP achieved predictive performance (measured by SMSE and MSLL) comparable to fully connected models but with significantly higher efficiency. The model identified that the complex walking motion could be explained by as few as two primary latent driving functions.

Gene Expression

The model was applied to the "p53" dataset, which monitors the response of five genes to the p53 tumor suppressor protein. The framework correctly identified that a single latent force was responsible for the observed expression patterns across all five genes. This result aligns with established biological knowledge, demonstrating the model's ability to uncover true underlying mechanisms without prior manual tuning.

Discussion and Limitations

While the IBP provides a robust mechanism for model selection, the authors note certain practical considerations:

  • Truncation: The variational implementation requires a truncation level ($Q$) to cap the maximum number of potential features.
  • Optimization: Like many variational methods, the model can be sensitive to initialization and may occasionally converge to local optima.
  • Scalability: Although more efficient than sampling, the computational cost remains a factor for very large datasets, suggesting that future work could integrate "sparse GP" approximations to further enhance speed.

Conclusion

By combining the non-parametric flexibility of the Indian Buffet Process with the physical grounding of process convolutions, this research provides a principled way to handle model selection in multi-output Gaussian processes. The ability to automatically learn the number and connectivity of latent forces makes this framework a valuable tool for interpreting complex systems in biology, robotics, and beyond.

Summary updated

Abstract

Multi-output Gaussian processes have received increasing attention during the\nlast few years as a natural mechanism to extend the powerful flexibility of\nGaussian processes to the setup of multiple output variables. The key point\nhere is the ability to design kernel functions that allow exploiting the\ncorrelations between the outputs while fulfilling the positive definiteness\nrequisite for the covariance function. Alternatives to construct these\ncovariance functions are the linear model of coregionalization and process\nconvolutions. Each of these methods demand the specification of the number of\nlatent Gaussian process used to build the covariance function for the outputs.\nWe propose in this paper, the use of an Indian Buffet process as a way to\nperform model selection over the number of latent Gaussian processes. This type\nof model is particularly important in the context of latent force models, where\nthe latent forces are associated to physical quantities like protein profiles\nor latent forces in mechanical systems. We use variational inference to\nestimate posterior distributions over the variables involved, and show examples\nof the model performance over artificial data, a motion capture dataset, and a\ngene expression dataset.\n
BibTeX
@misc{guarnizo2015indian,
  author = {Cristian Guarnizo and Mauricio A. Álvarez},
  title = {{Indian Buffet process for model selection in convolved multiple-output\textbackslash\{\}n Gaussian processes}},
  year = {2015},
  eprint = {1503.06432},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1503.06432},
}
Figure