Evaluation of the linkage disequilibrium method for estimating effective population size

Abstract

Data on linkage disequilibrium at unlinked loci provide an estimate of the inbreeding effective population size of the parental generation of the sampled cohort. The inbreeding effective population size, Ne, is the reciprocal of the probability that two gametes, selected at random without replacement from those that produced the sampled cohort, derive from the same parent. Effective population size is an important parameter for measuring the rate of inbreeding in a population. We detail the construction of the linkage disequilibrium estimator of Ne, and evaluate its performance by simulation. We simulate populations which are dioecious and non-selfing. We use the simulations to examine the effects of several types of deviation from ideal population conditions, and of sample size, genotyping errors, number of loci typed, and polymorphic loci. We find substantial bias in the Ne estimator when there have been recent fluctuations in census population size, when the index of breeding variability is greater than one, and when the ratio of sample size to effective population size differs substantially from one. Due to high variability, estimators that have low bias for the reciprocal of Ne can present substantial bias when used as estimators of Ne itself. We consider a recent small sample size bias correction proposed for the method, and find that it improves bias in the reciprocal, but at the expense of increased bias for Ne. The improvements in the bias of the reciprocal are usually small, but are substantial when sample size is much less than Ne, while the increase in bias for Ne is often substantial. We test the method on two exhaustively sampled rat populations, and find it performs as expected from simulation. For practitioners, we recommend that resources are spent first in ensuring that the sample size is likely to be greater than the effective population size, and only then that the number of loci is increased to improve the precision of the estimate.