Day24 Statistics Review (3)

📚 The Start of Recording TIL Summer ‘24 : Why Did I Start This Project? 🚀

🔔 Don't Miss: Continous Records of TIL '25 (Still On-Going!) 🌟

✨ A New Chapter in TIL '25: Reflecting, Growing, and Moving Forward 📖

Test Statistics (Z-Test, t-Test, and Chi-Squared Test)

(Ace the Data Science Interview: 201 Real Interview Questions Asked By FAANG, Tech Startups, & Wall Street)

Hypothesis Testing

It’s important to understand that hypothesis testing forms the foundation of A/B testing. In A/B testing, different versions of a feature are presented to various users in a sample, and each variant is evaluated to see if there was an improvement in the key engagement metrics.

Example of Hypothesis Testing

(https://www.investopedia.com/terms/h/hypothesistesting.asp)

If an individual wants to determine whether a penny has a 50% chance of landing on heads, they would set up a statistical test. The null hypothesis in this case would be that the probability of landing on heads is 50%, represented as $H_0 : P = 0.5$. The alternative hypothesis ($H_1$) would be that the probability is not 50%, indicated by striking through the equal sign in the null hypothesis.

A random sample of 100 coin flips is taken to conduct the test. If the sample yields 40 heads and 60 tails, the analyst would reject the null hypothesis in favor of the alternative hypothesis, concluding that a penny does not have a 50% chance of landing on heads.

However, if the sample shows 48 heads and 52 tails, it could still be consistent with a fair coin. In this case, where the null hypothesis is “accepted,” the analyst would attribute the difference between the expected and observed results to chance alone.

Test Statistics

A test statistic is a numerical summary used to determine whether the null or alternative hypothesis should be accepted as correct. Specifically, it assumes that the parameter of interest follows a particular sampling distribution under the null hypothesis.

For instance, the number of heads in a series of coin flips may follow a binomial distribution. However, with a sufficiently large sample size, the sampling distribution should be approximately normally distributed. Therefore, the sampling distribution for the total number of heads in a large series of coin flips would be considered normally distributed.

Here are several variations in test statistics and their distributions:

1. Z-test: This assumes that the test statistic follows a normal distribution under the null hypothesis.
2. t-test: This uses a student’s t-distribution instead of a normal distribution.
3. Chi-squared test: This is used to assess goodness of fit and to check whether two categorical variables are independent.

Z-Test

Typically, the Z-test is utilized when dealing with a large sample size to invoke the Central Limit Theorem or when the population variance is known. On the other hand, a t-test is used with a small sample size or when the population variance is unknown. The Z-test for a population mean is formulated as follows in the case where the population variance ($\sigma^2$) is known.

$z = \frac{\bar{x} - \mu_0}{\frac{\sigma}{\sqrt{n}}} \sim N(0,1)$

t-Test

The t-test is similar to the Z-test, but it uses the sample variance $s^2$ instead of the population variance. The t-test is determined by the degrees of freedom, which represent the number of independent observations in a dataset and are denoted by $n-1$.

$t=\frac{\bar{x}-\mu_0}{\frac{s}{\sqrt{n}}}\sim t_{n-1}$
where $s^2 = \frac{\sum_{i=1}^n(x_i-\bar{x})^2}{n-1}$

As mentioned, the t-distribution resembles the normal distribution but has larger tails, meaning extreme events occur more frequently than expected by the modeled distribution. This phenomenon is common, particularly in economics and Earth science.

Chi-Squared Test

The Chi-squared test statistic is utilized to evaluate goodness of fit and is calculated in the following.

$\chi^2 = \sum_i\frac{(O_i-E_i)^2}{E_i}$

where $O_i$ is the observed value and $E_i$ is the expected value. The degrees of freedom for the chi-squared test statistic depends on the number of categories in the distribution.

We can use a chi-squared test to determine if two categorical variables are independent. Start by creating a contingency table with one variable’s values as rows and the other as columns, then look for intersections in the table. This test uses the same chi-squared test statistic as mentioned above.