Performance and Limitations of Likelihood based Information Criteria and Leave-one-out Cross-validation Approximation Methods

Abstract

Model assessment, in the Bayesian context, involves evaluation of the goodness-of-fit and the comparison of several alternative candidate models for predictive accuracy and improvements. In posterior predictive checks, the data simulated under the fitted model is compared with the actual data. Predictive model accuracy is estimated using information criteria such as Akaike information criterion (AIC), Bayesian information criterion (BIC), Deviance information criterion (DIC), and Watanabe-Akaike information criterion (WAIC). The goal of an information criterion is to obtain an unbiased measure of out-of-sample prediction error. Since posterior checks use the data twice; once for model estimation and once for testing, a bias correction which penalises the model complexity is incorporated in these criteria. Cross-validation (CV) is another method used for examining out-of-sample prediction accuracy. Leave-one-out or exact cross-validation (LOO-CV) is the most computationally expensive variant among the other CV methods, as it fits as many models as the number of observations. Importance sampling (IS), truncated importance sampling (TIS) and Pareto-smoothed importance sampling (PSIS) are generally used as approximations to the exact LOO-CV and utilise the existing Makov Chain Monte Carlo (MCMC) results avoiding expensive computational issues. Although, information criteria and LOO are unable to reflect the goodness-of-fit in absolute sense, the differences can be used to measure the relative performance of the models of interest. However, the use of these measures is only valid under specific circumstances. In this paper we illustrate the limitations of the information criteria in practical model comparison problems with respect to an example from forensic statistics. In addition, the relationships among LOO approximation methods and WAIC with their limitations are discussed. Finally, we provide some useful recommendations that may help in practical model comparisons with these methods. Key words: information criteria, cross-validation, importance sampling, predictive accuracy