Fast Direct Methods for Gaussian Processes
Abstract
A number of problems in probability and statistics can be addressed using the multivariate normal (Gaussian) distribution. In the onedimensional case, computing the probability for a given mean and variance simply requires the evaluation of the corresponding Gaussian density. In the ndimensional setting, however, it requires the inversion of an n × n covariance matrix, C, as well as the evaluation of its determinant, \det(C). In many cases, such as regression using Gaussian processes, the covariance matrix is of the form C = σ^2 I + K, where K is computed using a specified covariance kernel which depends on the data and additional parameters (hyperparameters). The matrix C is typically dense, causing standard direct methods for inversion and determinant evaluation to require O(n^3) work. This cost is prohibitive for largescale modeling. Here, we show that for the most commonly used covariance functions, the matrix C can be hierarchically factored into a product of block lowrank updates of the identity matrix, yielding an O (n log^2 n) algorithm for inversion. More importantly, we show that this factorization enables the evaluation of the determinant \det(C), permitting the direct calculation of probabilities in high dimensions under fairly broad assumptions on the kernel defining K. Our fast algorithm brings many problems in marginalization and the adaptation of hyperparameters within practical reach using a single CPU core. The combination of nearly optimal scaling in terms of problem size with highperformance computing resources will permit the modeling of previously intractable problems. We illustrate the performance of the scheme on standard covariance kernels.
 Publication:

IEEE Transactions on Pattern Analysis and Machine Intelligence
 Pub Date:
 June 2015
 DOI:
 10.1109/TPAMI.2015.2448083
 arXiv:
 arXiv:1403.6015
 Bibcode:
 2015ITPAM..38..252A
 Keywords:

 Mathematics  Numerical Analysis;
 Astrophysics  Instrumentation and Methods for Astrophysics;
 Mathematics  Statistics Theory;
 Mathematics  Numerical Analysis;
 Astrophysics  Instrumentation and Methods for Astrophysics;
 Mathematics  Statistics Theory