Probability

Conditional Probability

• Conditional probability → Probability of event A given event B → Bayes’ rule → $P(A|B)=\frac{P(A\cap B)}{P(B)}=\frac{P(B|A)P(A)}{P(B)}$

  • $P(A)$ → prior

  • $P(B|A)$ → likelihood

  • $P(A\cap B)$ → probability that both events $A$ and $B$ occur.

  • $P(A)$ → posterior

  • Conditional probability is the probability of an event occurring given that another event has already occurred. It's a way of updating our beliefs about the likelihood of an event based on new information.

  • When $P(A|B)=P(A)$ then $A$ and $B$ are independent.

  • Note → It’s possible for $A$ and $B$ to be conditionally independent given the occurrence of another event $C$ → $P(A\cap B|C)=P(A|C)P(B|C)$ → that means, given that $C$ has occurred, knowing that $B$ has also occurred tells us nothing about (provides no additional knowledge) the probability of $A$ having occurred → However, conditional independence doesn’t necessarily mean $A$ and $B$ are independent → In other words, conditional independence does not imply unconditional independence → In other words, conditional independence is a concept where $A$ and $B$ are independent of each other given a third event $C$ → the occurrence of $C$ makes $A$ and $B$ independent.

    • Example:
      • $A$ → represents whether the ground is wet
      • $B$ → represent whether people are carrying umbrellas
      • $C$ → represents whether it’s raining
      • Given $C$, $A$ and $B$ might be conditionally independent because whether the ground is wet given it is raining does not affect whether people are carrying umbrellas given it is raining.
      • However, without knowing $C$, $A$ and $B$ are not independent. If you know the ground is wet, it might increase the likelihood that people are carrying umbrellas, and vice versa.
    • Application:
      • In Bayesian networks, conditional independence is fundamental in simplifying the representation of the joint probability distribution of the variables, allowing for efficient computation of probabilities and inferences.
      • In ML models like Naive Bayes classifiers and Hidden Markov Models, conditional independence assumptions simplify the model and the computations required for training and prediction. These assumptions enable the use of algorithms that are computationally efficient and scalable to large datasets.
      • In the context of regression and causal inference, conditional independence is used to understand the relationships between variables. It helps in identifying confounders, mediators, and colliders, which are essential in determining the causal effects and in controlling for variables when estimating these effects.
      • Conditional independence is crucial in the study of causality, where researchers aim to understand and establish cause-and-effect relationships between variables. Techniques such as the back-door criterion and instrumental variables use conditional independence to identify valid causal pathways and to control for confounding variables.
      • Medical testing for rare disease → it may be misleading to simply diagnose someone as having a disease, even if the test for the disease is considered “very accurate”, without knowing the test’s base rate for accuracy.

    <aside> 💡 INTERVIEW TIP If other information is available and you are asked to calculate a probability, you should always consider using Bayes’ rule.

    </aside>

Law of Total Probability

• Assume we have several disjoint events within $B$ having occurred. Using the law of total probability, we can break down the probability of $A$ as follows:

  $$ P(A)=P(A|B_1)P(B_1)+\dots+P(A|B_n)P(B_n)=\sum_{i=1}^n P(A|B_i)P(B_i) $$

  <aside> 💡 DEFINITION: DISJOINT EVENTS

  In probability, disjoint events, also known as mutually exclusive events, are events that cannot occur at the same time. This means that they have no outcomes in common. For example, the result of a coin toss can be either heads or tails, but not both, so these are disjoint events.

  </aside>

  • Law of total probability provides a handy way of decomposing the $P(A)$ into the weighted sum of conditional probabilities.
  • Example: One common example is the probability that a customer makes a purchase, conditional on which customer segment that customer falls within.

Counting

• Generally, there are two forms of counting:
  • Permutation → order of selection matters → $P(n,k)=\frac{n!}{(n-k)!}$
    • Circular permutation → the number of ways to arrange $n$ distinct items in a circle is: $(n-1)!$
      • Example: How many ways can five people sit around a lunch table?
  • Combination → order of selection doesn’t matter → $C(n,k)= \frac{n!}{k!(n-k)!}$
• Example: What is the likelihood that I draw four cards of the same suit?
  • Solution 1
    • There are $C(52,4)$ ways to pick 4 cards from a deck of 52.
    • There are $C(13,4)$ ways to pick 4 cards of the same suit.
    • Answer: $4 \times \frac{C(13,4)}{C(52,4)}$
  • Solution 2
    • There is a $\frac{1}{4}$ chance of picking a particular suit.
    • Given the suit (and the 51 cards left), the probability of next card being from the same suit is → $\frac{12}{51}$
    • Similarly, the probability of picking the third and fourth card from the same suit is $\frac{11}{50}$ and $\frac{10}{49}$, respectively.
    • Answer: $\frac{1}{4} \times \frac{12}{51} \times \frac{11}{50} \times \frac{10}{49}$
  • Which is the right solution?!
• Application:
  • Making up password (permutation)
  • Choosing restaurants nearby (combination)

Random Variables

• A random variable , $X$, is a quantity with an associated probability distribution.
• Random variable can be either:
  • Discrete → i.e., have a countable range
    • $X$ can take on particular values with a particular probability
  • Continuous
    • The probability of a particular value of $x$ is not measurable → instead, a “probability mass” per unit of length around $x$ can be measured (imagine a small interval of $x$ and $x + \delta$).
• The probability distribution ,$x:f_X(x)$, of a random variable:
  • Discrete → Probability Mass Distribution (PMF)
  • Continuous → Probability Density Function (PDF)
  • Probabilities in probability distributions must be non-negative and must sum (or integrate) to 1.
    • Discrete → $\sum_{x \in X}f_X(x)=1$
    • Continuous → $\int_{-\infty}^{\infty}f_X(x)dx=1$
• The Cumulative Probability Function (CDF) is often used in practice → $F_X(x)=p(X\le x)$
  • Discrete → $F_x(x)=\sum_{k\le x}p(k)$
  • Continuous → $F_X(x)=\int_{-\infty}^{\infty}p(y)dy$
  • CDF is non-negative and monotonically increasing.

Joint, Marginal, and Conditional Probability

• Random variables are often analyzed with respect to each other → Joint PDFs.

• Random variables $X$ and $Y$ varying over a two-dimensional space → the integration of the joint PDF yields the following

  $$ \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} f_{X,Y}(x,y)dxdy = 1 $$