Large-scale Inference Using Non-parametric Mixtures

Abstract

A new approach to solving large-scale problems using non-parametric mixtures is proposed. The new approach not only takes the advantage of the great flexibility in nonparametric mixture methods and the fast computation of the non-parametric maximum likelihood estimate of a mixing distribution using the constrained Newton method proposed by Wang (2007), but it is also able to make use of the common features among observations in order for a more efficient and accurate estimation. This new approach is then applied to several practical fields. In the context of multiple hypothesis testing, a new method is proposed to compute the NPMLE given a fixed probability at the location of the null effect, and then is extended to estimate the proportion of null effects based on various threshold functions. Distance-based counterparts under the Cramér-von Mises and the Anderson-Darling distance are also introduced. Furthermore, modifications are also made to the likelihood- and distancebased methods, and hence they can be used in large-scale computation. A new procedure that controls the false discovery rate, based on the estimated null proportion and its estimated mixing distribution, is also introduced. Numerical studies show that the estimators of the null proportion using the non-parametric maximum likelihood estimators and their minimum distance counterparts (the non-parametric minimum distance estimators) perform well and the proposed controlling procedure makes more rejections than existing methods given a pre-specified level. A new method based on this new approach is also proposed for covariance matrix estimation. A new covariance matrix estimator is constructed using the empirical Bayes on the sample correlation coefficients or the transformed sample correlation coefficients. The estimated density required by the proposed estimator can be computed using the non-parametric maximum likelihood estimators or the non-parametric minimum distance estimators. Estimation under the sparsity assumption is also discussed. The numerical studies using the simulated and the real world datasets all suggest that the proposed covariance matrix estimator performs well and can be applied to a wide range of covariance structures.