Conditional Probabilities are Probabilities

When we condition on an event $E$, we update our beliefs to be consistent with this knowledge, effectively putting ourselves in a universe where we know that $E$ occurred. Within our new universe, however, the laws of probability operate just as before. Conditional probability satisfies all the properties of probability! Therefore, any of the results we have derived about probability are still valid if we replace all unconditional probabilities with probabilities conditional on $E$.

For example, here are conditional forms of Bayes' rule and the law of total probability. These are obtained by taking the ordinary forms of Bayes' rule and LOTP and adding $E$ to the right of the vertical bar everywhere.

Theorem: Bayes' Rule with Extra Conditioning

Provided that $P(A \cap E) > 0$ and $P(B \cap E) > 0$, we have

$$P(A|B,E) = \frac{P(B|A,E) P(A|E)}{P(B|E)}.$$

Theorem: LOTP with Extra Conditioning

Let $A_1,\dots,A_n$ be a partition of $S$. Provided that $P(A_i \cap E) > 0$ for all $i$, we have

$$P(B|E) = \sum_{i=1}^n P(B|A_i, E) P(A_i|E).$$

Independence of Events

We have now seen several examples where conditioning on one event changes our beliefs about the probability of another event. The situation where events provide no information about each other is called independence.

Definition: Independence of Two Events

Events $A$ and $B$ are independent if

$$P(A \cap B) = P(A) P(B).$$

If $P(A) > 0$ and $P(B) > 0$, then this is equivalent to

$$P(A|B) = P(A),$$

and also equivalent to $P(B|A) = P(B).$

In words, two events are independent if we can obtain the probability of their intersection by multiplying their individual probabilities. Alternatively, $A$ and $B$ are independent if learning that $B$ occurred gives us no information that would change our probabilities for $A$ occurring (and vice versa).

Note that independence is a symmetric relation: if $A$ is independent of $B$, then $B$ is independent of $A$.

Independence is completely different from disjointness. If $A$ and $B$ are disjoint, then $P(A \cap B) = 0$, so disjoint events can be independent only if $P(A) = 0$ or $P(B) = 0$. Knowing that $A$ occurs tells us that $B$ definitely did not occur, so $A$ clearly conveys information about $B$.

We also often need to talk about independence of three or more events.

Definition: Independence of Three Events

Events $A$, $B$, and $C$ are said to be independent if all of the following equations hold:

$$\begin{align*} P(A \cap B) &= P(A) P(B), \\ P(A \cap C) &= P(A) P(C), \\ P(B \cap C) &= P(B) P(C), \\ P(A \cap B \cap C) &= P(A) P(B) P(C). \end{align*}$$

If the first three conditions hold, we say that $A$, $B$, and $C$ are pairwise independent. Pairwise independence does not imply independence: it is possible that just learning about $A$ or just learning about $B$ is of no use in predicting whether $C$ occurred, but learning that both $A$ and $B$ occurred could still be highly relevant for $C$. Here is a simple example of this distinction.

Example Pairwise Independence doesn't Imply Independence

We can define independence of any number of events similarly. Intuitively, the idea is that knowing what happened with any particular subset of the events gives us no information about what happened with the events not in that subset.

Conditional independence is defined analogously to independence.

Definition: Conditional Independence

Events $A$ and $B$ are said to be conditionally independent given event $E$ if $P(A \cap B|E) = P(A|E) P(B|E)$.

Example Conditional Independence doesn't Imply Independence

Example Independence doesn't Imply Conditional Independence