Day22 Statistics Review (1)

📚 The Start of Recording TIL Summer ‘24 : Why Did I Start This Project? 🚀

🔔 Don't Miss: Continous Records of TIL '25 (Still On-Going!) 🌟

✨ A New Chapter in TIL '25: Reflecting, Growing, and Moving Forward 📖

Properties of Random Variable

(Ace the Data Science Interview: 201 Real Interview Questions Asked By FAANG, Tech Startups, & Wall Street)

Statistics is a fundamental aspect of any data scientist’s toolbox. Many components of a data science pipeline rely on statistical principles, making a strong understanding of foundational statistics essential.

We start with topics like the Central Limit Theorem and the Law of Large Numbers and then progress to the concepts underlying hypothesis testing, particularly p-values and confidence intervals, as well as type I and type II errors and their interpretations.

The following properties hold for any given random variable $X$. We assume $X$ is continuous, but these properties also hold for discrete random variables.

The expectation, also known as the average value or mean, of a random variable is calculated by taking the integral of the value of X multiplied by its probability density function (PDF) $f_X(x)$ :

$\mu = \mathbb{E}[X] = \int_{-\infty}^{\infty} x f_X(x) \, dx$

and the variance is given by:

$\mathrm{Var}(X) = \mathbb{E} \left[ (X - \mathbb{E}[X])^2 \right] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$

The variance is always non-negative, and its square root is known as the standard deviation, which is widely used in statistics.

$\sigma = \sqrt{\mathrm{Var}(X)} = \sqrt{\mathbb{E} \left[ (X - \mathbb{E}[X])^2 \right]} = \sqrt{\mathbb{E}[X^2] - (\mathbb{E}[X])^2}$

The conditional expectations of both the expectation and variance are as follows. For instance, let’s consider the case for the conditional expectation of $X$, given that $Y=y$.

$ \mathbb{E}[X | Y = y] = \int_{-\infty}^{\infty} x f_{X|Y}(x|y) \, dx $

For any given pair of random variables $X$ and $Y$, the covariance is a linear measure of the relationship between the two variables.

$\mathrm{Cov}(X, Y) = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y]$

and the normalization of covariance, represented by the Greek letter $\rho$, is the correlation between $X$ and $Y$.

$\rho(X, Y) = \frac{\mathrm{Cov}(X, Y)}{\sqrt{\mathrm{Var}(X)\mathrm{Var}(Y)}}$

All of these properties are basic concepts for DS statistics, so it helps to be able to understand the mathematical details behind each and walk through an example for each.

For example, if we assume $X$ follows a Uniform distribution on the interval $[a, b]$, then we have the following:

$f_X(x) = \frac{1}{b-a}$

Therefore, the expectation of $X$ is:

$$ \mathbb{E}[X] = \int_a^b x f_X(x) \, dx = \int_a^b x \frac{1}{b-a} \, dx = \frac{x^2}{2(b-a)} \bigg|_a^b = \frac{a+b}{2} $$

We don’t have to memorize the derivations of all the various probability distributions, but we should be able to derive them when necessary. We need to understand the formulas provided above and be able to apply them to common probability distributions such as the exponential or uniform distribution.