Variance

One important application of LOTUS is for finding the* variance* of a random variable. Like expected value, variance is a single-number summary of the distribution of a random variable. While the expected value tells us the center of mass of a distribution, the variance tells us how spread out the distribution is.

Definition: Variance and Standard Deviation

The variance of an r.v. $X$ is

$$\textrm{Var}(X) = E(X-EX)^2.$$

The square root of the variance is called the standard deviation (SD):

$$\textrm{SD}(X) = \sqrt{\textrm{Var}(X)}.$$

Recall that when we write $E(X-EX)^2$, we mean the expectation of the random variable $(X-EX)^2$, not $(E(X-EX))^2$ (which is $0$ by linearity).

The variance of $X$ measures how far $X$ is from its mean on average, but instead of simply taking the average difference between $X$ and its mean $EX$, we take the average squared difference. To see why, note that the average deviation from the mean, $E(X-EX)$, always equals 0 by linearity; positive and negative deviations cancel each other out. By squaring the deviations, we ensure that both positive and negative deviations contribute to the overall variability. However, because variance is an average squared distance, it has the wrong units: if $X$ is in dollars, $\textrm{Var}(X)$ is in squared dollars. To get back to our original units, we take the square root; this gives us the standard deviation.

One might wonder why variance isn't defined as $E|X-EX|$, which would achieve the goal of counting both positive and negative deviations while maintaining the same units as $X$. This measure of variability isn't nearly as popular as $E(X-EX)^2$, for a variety of reasons. The absolute value function isn't differentiable at 0, so it doesn't have as nice properties as the squaring function. Squared distances are also connected to geometry via the distance formula and the Pythagorean theorem, which turn out to have corresponding statistical interpretations.

An equivalent expression for variance is $E(X-EX)^2$. This formula is often easier to work with when doing actual calculations. Since this is the variance formula we will use over and over again, we state it as its own theorem.

Theorem

For any r.v. $X$,

$$\textrm{Var}(X) = E(X^2) - (EX)^2.$$

Proof: Let $\mu = EX$. Expand $(X-\mu)^2$ and use linearity:

Variance has the following properties. The first two are easily verified from the definition, the third will be addressed in a later chapter, and the last one is proven just after stating it.

To prove the last property, note that $\textrm{Var}(X)$ is the expectation of the nonnegative r.v. $(X-EX)^2$, so $\textrm{Var}(X) \geq 0$. If $P(X=a)=1$ for some constant $a$, then $E(X)=a$ and $E(X^2)=a^2$, so $\textrm{Var}(X)=0$.

Conversely, suppose that $\textrm{Var}(X) = 0$. Then $E(X-EX)^2 = 0$, which shows that $(X-EX)^2=0$ has probability $1$, which in turn shows that $X$ equals its mean with probability $1$.

Example Geometric and Negative Binomial Variance

Example Binomial Variance