Saddlepoint methods in neural networks

Abstract

Saddlepoint methods have found increased application in recent years. They provide fast and accurate approximations to some of the most important integrals encountered in Statistics. In addition, when used to approximate probability distributions, they yield accurate tail probabilities. Competing methods, such as those based on stochastic simulation, are much slower and typically yield very poor approximations to the tail of a distribution. However, as we shall see in the forthcoming chapters, saddlepoint methods routinely provide virtually exact approximations to many quantities of interest besides probability distributions. In the first chapter, Laplace's method is used to perform marginal inference in Bayesian neural networks. Accurate approximations for Bayes factors for model choice about the number of nonlinear sigmoidal terms: predictive densities for a future observable; Bayes estimates for the nonlinear regression function; and the marginal densities are given. Important use is made of the inherent partial linearity of the regression function and the lack of identifiability. The choice of prior and the use of an alternative sigmoidal lead to posterior invariance in the nonlinear parameter which is discussed in connection with the lack of sigmoidal identifiability. The accuracy of the Laplace approximations is illustrated in the context of two nonlinear data sets: a nonlinear regression model and a nonlinear autoregressive time series. In chapter two, saddlepoint approximations and Laplace's method are used to study classification in the stochastic Hopfield model (SHM). First, the methodology is developed to provide saddlepoint approximations to classification time distributions. Secondly, saddlepoint approximations to the stationary distribution of the Hopfield Markov chain, the Markov chain underlying the SHM's classification process, are presented. These approximations are particularly difficult to obtain since this stationary distribution has an intractable moment generating function which we approximate with Laplace's method. Lastly, a characterization of the set of possible absorbing states of the Hopfield Markov chain for the deterministic Hopfield model, a forerunner of the SHM is provided. All of our contributions are a result of the lumpability of the Hopfield Markov chain which is rigorously derived and proven. The accuracy of the saddlepoint methods, in the above classification problems, is demonstrated on a SHM with 210 = 1024 states which reduced to a model with a mere 64 states via lumping.