Estimating and interpreting F[subscript ST]: The impact of rare variants

Abstract

In a pair of seminal papers, Sewall Wright and Gustave Malécot introduced F[subscript ST] as a measure of structure in natural populations. In the decades that followed, a number of papers provided differing definitions, estimation methods, and interpretations beyond Wright's. While this diversity in methods has enabled many studies in genetics, it has also introduced confusion regarding how to estimate F[subscript ST] from available data. Considering this confusion, wide variation in published estimates ofF[subscript ST] for pairs of HapMap populations is a cause for concern. These estimates changed—in some cases more than twofold—when comparing estimates from genotyping arrays to those from sequence data. Indeed, changes in F[subscript ST] from sequencing data might be expected due to population genetic factors affecting rare variants. While rare variants do influence the result, we show that this is largely through differences in estimation methods. Correcting for this yields estimates of F[subscript ST] that are much more concordant between sequence and genotype data. These differences relate to three specific issues: (1) estimating F[subscript ST] for a single SNP, (2) combining estimates of F[subscript ST] across multiple SNPs, and (3) selecting the set of SNPs used in the computation. Changes in each of these aspects of estimation may result in F[subscript ST] estimates that are highly divergent from one another. Here, we clarify these issues and propose solutions.