API

Extended Kalman filter (EKF)

• ekf.dense_fisher applies an online Bayesian update based on a Taylor approximation of the log-likelihood. Uses the empirical Fisher information matrix as a positive-definite alternative to the Hessian. Natural gradient descent equivalence following Ollivier, 2019.
• ekf.diag_fisher same as ekf.dense_fisher but uses the diagonal of the empirical Fisher information matrix instead.

Laplace approximation

• laplace.dense_fisher calculates the empirical Fisher information matrix and uses it to approximate the posterior precision, i.e. a Laplace approximation.
• laplace.dense_ggn calculates the Generalised Gauss-Newton matrix which is equivalent to the non-empirical Fisher in most neural network settings - see Martens, 2020.
• laplace.dense_hessian calculates the Hessian of the negative log posterior.
• laplace.diag_fisher same as laplace.dense_fisher but uses the diagonal of the empirical Fisher information matrix instead.
• laplace.diag_ggn same as laplace.dense_ggn but uses the diagonal of the Generalised Gauss-Newton matrix instead.

All Laplace transforms leave the parameters unmodified. Comprehensive details on Laplace approximations can be found in Daxberger et al, 2021.

Stochastic gradient Markov chain Monte Carlo (SGMCMC)

• sgmcmc.sgld implements stochastic gradient Langevin dynamics (SGLD) from Welling and Teh, 2011.
• sgmcmc.sghmc implements the stochastic gradient Hamiltonian Monte Carlo (SGHMC) algorithm from Chen et al, 2014 (without momenta resampling).
• sgmcmc.sgnht implements the stochastic gradient Nosé-Hoover thermostat (SGNHT) algorithm from Ding et al, 2014, (SGHMC with adaptive friction coefficient).
• sgmcmc.baoa implements the BAOA integrator for SGHMC from Leimkuhler and Matthews, 2015 - p271.

For an overview and unifying framework for SGMCMC methods, see Ma et al, 2015.

Variational inference (VI)

• vi.dense implements a Gaussian variational distribution. Expects a torchopt optimizer for handling the minimization of the NELBO. Also find vi.dense.nelbo for simply calculating the NELBO with respect to a log_posterior and Gaussian distribution.
• vi.diag same as vi.dense but uses the diagonal of the Gaussian variational distribution.

A review of variational inference can be found in Blei et al, 2017.

Optim

• optim wrapper for torch.optim optimizers within the unified posteriors API that allows for easy swapping with UQ methods.

TorchOpt

• torchopt wrapper for torchopt optimizers within the unified posteriors API that allows for easy swapping with UQ methods.