Abstract
Gaussian Processes (GPs) are powerful tools for modeling functions and quantifying uncertainty, but they traditionally struggle to scale when predicting high-dimensional, multi-output data. We propose HOGPR (High-Order Gaussian Process Regression), a model that tensorizes outputs and exploits Kronecker product algebra to perform exact, non-linear GP inference on up to a million outputs. By introducing latent coordinate features to capture complex correlations, HOGPR bridges the gap between highly expressive but unscalable GPs and scalable but overly simplistic linear methods.
The Challenge of High-Dimensional Outputs
Many scientific and engineering applications require learning mappings from low-dimensional inputs to massively high-dimensional outputs. For instance, in computational physics, a small set of boundary conditions might dictate a complex pressure field represented by millions of spatial points. Standard multi-output GPs are mathematically intractable for such tasks, as their computational complexity scales cubically—$O((Nd)^3)$—with the number of outputs $d$.
To circumvent this, researchers often resort to low-rank approximations like PCA-GP. However, these methods rely on linear assumptions that are often too rigid to capture the complex, non-linear correlations present in real-world physical fields, often resulting in blurred or inaccurate predictions.
Methodology: Tensorization and Kronecker Algebra
HOGPR achieves exact inference at scale through a structural assumption that enables efficient computation:
- Tensorization: Outputs are reshaped into a multi-dimensional tensor (e.g., a 3D spatial grid) rather than a flat vector.
- Latent Coordinate Features: The model learns latent features for each dimension of the tensor to capture underlying correlations between different outputs.
- Multilinear Tucker Operator: By combining original inputs with these latent features, HOGPR can capture highly non-linear relationships.
- Scalability: This tensor structure factors the massive covariance matrix into a Kronecker product of smaller matrices. This allows HOGPR to compute exact marginal likelihoods in $O(N^2 d)$ time—scaling linearly with the number of outputs.
Quantitative Performance and Stability
HOGPR was evaluated against several baselines, including Sparse Convolved GPs (SCGPR) and various low-rank methods (PCA-GPR, IsoMap-GPR). Across small-scale and massive datasets, HOGPR consistently achieved lower Root-Mean-Squared-Error (RMSE) and demonstrated superior numerical stability. While SCGPR often failed due to memory constraints as training sizes increased, HOGPR remained robust.
Visual Fidelity in Topology Design
In tasks like topology structure design, the advantage of HOGPR's non-linear modeling becomes visually apparent. Methods like PCA-GPR often blur local details because they rely on global linear bases. In contrast, HOGPR preserves sharp, fine-grained local structures, producing predictions that closely match the ground truth.
Conclusion
HOGPR demonstrates that exact GP inference is possible even for million-dimensional outputs. By leveraging tensor algebra and latent coordinate modeling, it provides a scalable surrogate modeling framework that maintains the expressivity of non-linear kernels without the computational burden of traditional multi-output GPs.
Summary updated