GaussianProcessTrainer
"Gaussian Processes for Machine Learning" Rasmussen and Williams (2005) provides a well written discussion of Gaussian Processes, and its reading is highly encouraged. Chapters 1-5 cover the topics presented here with far greater detail, depth, and rigor. Furthermore, for a detailed overview on Gaussian Processes that model multiple outputs together (multi-output Gaussian Process or MOGP) we refer the reader to Liu et al. (2018).
The documentation here is meant to give some practical insight for users to begin creating surrogate models with Gaussian Processes.
Given a set of inputs for which we have made observations of the correspond outputs using the system (). Given another set of inputs we wish to predict the associated outputs without evaluation of , which is presumed costly.
Parameter Covariance
In overly simplistic terms, Gaussian Process Modeling is driven by the idea that trials which are "close" in their input parameter space will be "close" in their output space. Closeness in the parameter space is driven by the covariance function (also called a kernel function, not to be confused with a MOOSE Framework kernel). This covariance function is used to generate a covariance matrix between the complete set of parameters , which can then be interpreted block-wise as various covariance matrices between and .
The Gaussian Process Model consists of an infinite collection of functions, all of which agree with the training/observation data. Importantly the collection has closed forms for 2nd order statistics (mean and variance). When used as a surrogate, the nominal value is chosen to be the mean value. The method can be broken down into two step: definition of the prior distribution then conditioning on observed data.
Gaussian processes
A Gaussian Process is a (potentially infinite) collection of random variables, such that the joint distribution of every finite selection of random variables from the collection is a Gaussian distribution.
(1)In an analogous way that a multivariate Gaussian is completely defined by its mean vector and its covariance matrix, a Gaussian Process is completely defined by its mean function and covariance function.
The (potentially) infinite number of random variables within the Gaussian Process correspond to the (potentially) infinite points in the parameter space our surrogate can be evaluated at.
Prior distribution:
We assume the observations (both training and testing) are pulled from an multivariate Gaussian distribution. The covariance matrix is the result of the choice of covariance function.
Note that and are a vector and matrix respectively, and are a result of the mean and covariance functions applied to the sample points.
Zero Mean Assumption: Discussions of Gaussian Process are typically presented under assumption that . This occurs without loss of generality since any sample can be made by subtracting the sample mean (or a variety of other preprocessing options). Note that in a training\testing paradigm, the testing data is unknown, so determination of what to use as is based on the information from the training data (or some other prior assumption).
Conditioning:
With the prior formed as above, conditioning on the available training data is performed. This alters the mean and variance to new values and , restricting the set of possible functions which agree with the training data.
When used as a surrogate, the nominal value is typically taken as the mean value, with providing variances which can be used to generate confidence intervals.
Notes on Multi-Output Gaussian Processes (MOGPs)
MOGPs model and predict the outputs which are vectors, each of size . For any input vector , the vector of outputs and the matrix of vectors is defined as:
(2)
where is of size and is of size . The matrix is vectorized and represented as with size . is modeled as a Gaussian distribution defined as described in Eq. (1).
In a multi-output Gaussian Process, captures covariances across the input variables and the vector of outputs and hence has size . can be modeled in several ways as discussed in (Liu et al., 2018; Alvarez et al., 2012). We will follow the linear model of coregionalization (LMC) which introduces latent functions with restrictons on the associated covariances.
Common Hyperparameters
While the only apparent decision in the above formulation is the choice of covariance function, most covariance functions will contain hyperparameters of some form which need to be selected in some manner. While each covariance function will have its own set of hyperparameters, a few hyperparameters of specific forms are present in many common covariance functions.
Length Factor or
Frequently Kernels consider the distance between two input parameters and . For system of only a single parameter this distance often takes the form of
In this form the factor set a relevant length scale for the distance measurements.
When multiple input parameters are to be considered, it may be advantageous to specify different length scales for each of the parameters, resulting in a vector . For example distance may be calculated as
When used with standardized parameters, can be interpreted in units of standard deviation for the relevant parameter.
Signal Variance
This serves as an overall scaling parameter. Given a covariance function (which is not a function of ), the multiplication of yields a new valid covariance function.
This multiplication can also be pulled out of the covariance matrix formation, and simply multiply the matrix formed by
Noise Variance
The represents noise in the collected data, and is as a additional factor on the variance terms (when ).
In the matrix representation this adds a factor of to diagonal of the noiseless matrix
Due to the addition of along the diagonal of the matrix, this hyperparameter can aid in the inversion of the covariance matrix. For this reason adding a small amount of may be preferable, even when you believe the data to be noise free.
Selected Covariance Functions
Table 1: Selected Covariance Functions
| Covariance Function | Description |
|---|---|
| SquaredExponentialCovariance | Also referred to as a radial basis function (RBF) this is a widely used, general purpose covariance function. Serves as a common starting point for many. Used for single-output GPs |
| ExponentialCovariance | A simple exponential covariance function. Used for single-output GPs |
| MaternHalfIntCovariance | Implementation of the Matern class of covariance function, where the parameter takes on half-integer values. Used for single-output GPs |
| LMC | Covariance function built using the linear model of coregionalization. Used for multi-output GPs |
Hyperparameter tuning options
The following options are available for tuning the hyperparameters:
Adaptive moment estimation (Adam)
Relies on the pseudocode provided in Kingma and Ba (2014). Adam permits stochastic optimization, wherein, a batch of the training data can be randomly chosen at each iteration.
The hyper-parameters of the GPs are inferred by optimizing the log-likelihood function. The MOGP log-likelihood function has a general form:
(3)
The optimization of GPs can be expensive. If there are training points each with outputs, each training iteration of Adam has a cost of . Adam permits using random training points during each iteration (mini-batches) which has a cost of .
The detailed documentation of this object is only available when Moose is compiled with Libtorch. For instructions on how to compile Moose with Libtorch, visit the general installation webpage or click here.
References
- M. A. Alvarez, L. Rosasco, and N. D. Lawrence.
Kernels for vector-valued functions: a review.
Foundations and Trends in Machine Learning, 4(3):195–266, 2012.[Export]
- D. P. Kingma and J. Ba.
Adam: a method for stochastic optimization.
arXiv:1412.6980 [cs.LG], 2014.
URL: https://doi.org/10.48550/arXiv.1412.6980.[Export]
- H. Liu, J. Cai, and Y. S. Ong.
Remarks on multi-output gaussian process regression.
Knowledge-Based Systems, 144:102–112, 2018.[Export]
- Carl Edward Rasmussen and Christopher K. I. Williams.
Gaussian Processes for Machine Learning.
The MIT Press, 2005.
ISBN 026218253X.[Export]