A Fully Natural Gradient Scheme for Improving Inference of the Heterogeneous Multioutput Gaussian Process Model | JoVE Visualize

A Fully Natural Gradient Scheme for Improving Inference of the Heterogeneous Multioutput Gaussian Process Model

Juan-Jose Giraldo, Mauricio A Alvarez

IEEE Transactions on Neural Networks and Learning Systems|May 24, 2021

View abstract on PubMed

Summary

This study introduces a novel natural gradient (NG) optimization for multioutput Gaussian processes (GPs) with heterogeneous outputs. The new method improves hyperparameter inference and achieves better performance than adaptive gradient methods.

Area of Science:

Machine Learning
Statistical Modeling
Computational Statistics

Background:

Multioutput Gaussian processes (GPs) are extended to model heterogeneous outputs, each with its own likelihood function.
A vector-valued GP prior models likelihood parameters as latent functions using a linear model of coregionalization (LMC) covariance.
Inducing points and stochastic variational inference (SVI) provide tractable variational bounds but face optimization challenges.

Purpose of the Study:

To develop an improved optimization scheme for heterogeneous multioutput GP models.
To address the burden on adaptive gradient methods caused by strong conditioning between variational parameters and hyperparameters.
To introduce an extension using convolution processes for latent functions and optimize it via natural gradient (NG) methods.

Main Methods:

Incorporation of an exploratory distribution over hyperparameters for joint inference with variational parameters.
Application of a fully natural gradient (NG) optimization scheme.
Extension of the heterogeneous model with latent functions drawn from convolution processes, made scalable via SVI and optimized with NG.

Main Results:

The proposed NG optimization scheme achieves better local optima and higher test performance rates compared to adaptive gradient methods for both LMC and convolution process models.
The convolutional model is demonstrated to be scalable using SVI and optimizable through the NG scheme.
Comparative performance analysis across toy and real datasets highlights the effectiveness of the novel optimization approach.

Conclusions:

The natural gradient optimization scheme offers a significant improvement for heterogeneous multioutput Gaussian processes, particularly with convolution process extensions.
The developed methods enhance model performance and scalability, providing a robust alternative to existing adaptive gradient techniques.
This work advances the field of multioutput GP modeling by providing more efficient and effective inference and optimization strategies.