Inbreeding coefficients

Now that we've found an expression for $\hat x_{12}$ we can also find expressions for $\hat x_{11}$ and $\hat x_{22}$. The complete set of equations for the genotype frequencies with partial selfing are:

$$\hat x_{11} = p^2 + \frac{\sigma pq}{2(1-\sigma/2)}$$ (18)

$$\hat x_{12} = 2pq - 2\left(\frac{\sigma pq}{2(1-\sigma/2)}\right)$$ (19)

$$\hat x_{22} = q^2 + \frac{\sigma pq}{2(1-\sigma/2)}$$ (20)

Notice that all of those equations have a term $\sigma/(2(1-\sigma/2))$. Let's call that $f$. Then we can save ourselves a little hassle by rewriting the above equations as:

$$\hat x_{11} = p^2 + fpq$$ (21)

$$\hat x_{12} = 2pq(1-f)$$ (22)

$$\hat x_{22} = q^2 + fpq$$ (23)

Now you're going to have to stare at this a little longer, but notice that $\hat x_{12}$ is the frequency of heterozygotes that we'd observe and $2pq$ is the frequency of heterozygotes we'd expect under Hardy-Weinberg in this population if we were able to observe the genotype and allele frequencies without error. So

$$1-f = \frac{\hat x_{12}}{2pq}$$ (24)

$$f = 1 - \frac{\hat x_{12}}{2pq}$$ (25)

$$= 1 - \frac{\hbox{observed heterozygosity}}% {\hbox{expected heterozygosity}}$$ (26)

$f$ is the inbreeding coefficient. When defined as 1 - (observed heterozygosity)/(expected heterozygosity) it can be used to measure the extent to which a particular population departs from Hardy-Weinberg expectations.⁴ When $f$ is defined in this way, I refer to it as the population inbreeding coefficient.

But $f$ can also be regarded as a function of a particular system of mating. With partial self-fertilization the population inbreeding coefficient when the population has reached equilibrium is $\sigma/(2(1-\sigma/2))$. When regarded as the inbreeding coefficient predicted by a particular system of mating, I refer to it as the equilibrium inbreeding coefficient.

We'll encounter at least two more definitions for $f$ once I've introduced ideas of identity by descent.