Limit Theorems (LLN & CLT) | Probability Theory

Core Concepts

Convergence in Distribution

Definition

Distribution functions $F_n$ converge weakly to $F$ , denoted $F_n \stackrel{w}{\to} F$ , if:

\lim_{n\to\infty}F_n(x)=F(x)

for all continuity points $x$ of $F(x)$ .

Lévy Continuity Theorem

\xi_n \stackrel{d}{\to} \xi \iff f_n(t) \to f(t) \text{ for all } t \in \mathbb{R}

Convergence in distribution is equivalent to convergence of characteristic functions.

Convergence in Probability

Definition

\xi_n \stackrel{P}{\to} \xi \iff \lim_{n\to\infty}P(|\xi_n-\xi|\geq\varepsilon)=0 \text{ for all } \varepsilon > 0

Convergence in probability is stronger than convergence in distribution.

Slutsky's Lemma: If $\xi_n \stackrel{d}{\to} \xi$ and $\eta_n \stackrel{P}{\to} c$ , then $\xi_n + \eta_n \stackrel{d}{\to} \xi + c$ , $\xi_n\eta_n \stackrel{d}{\to} c\xi$ , etc.

Law of Large Numbers

Weak Law of Large Numbers (WLLN)

Theorem:

\text{If } \{\xi_n\} \text{ i.i.d. with } E|\xi_1| < \infty, \text{ then } \frac{1}{n}\sum_{k=1}^n \xi_k \stackrel{P}{\to} E\xi_1

The sample mean converges in probability to the population mean.

Strong Law of Large Numbers (SLLN)

Kolmogorov SLLN:

\text{If } \{\xi_n\} \text{ i.i.d., then } \frac{1}{n}\sum_{k=1}^n \xi_k \stackrel{a.s.}{\to} E\xi_1 \iff E|\xi_1| < \infty

The sample mean converges almost surely to the population mean if and only if the expectation is finite.

Central Limit Theorem

Lindeberg-Lévy CLT

Theorem:

If $\{\xi_n\}$ are i.i.d. with $E\xi_1 = \mu$ and $0 < \text{Var}(\xi_1) = \sigma^2 < \infty$ , then:

\frac{\sum_{k=1}^n \xi_k - n\mu}{\sqrt{n}\sigma} \stackrel{d}{\to} N(0,1)

The normalized sum converges in distribution to standard normal distribution.

de Moivre-Laplace Theorem

Normal Approximation to Binomial:

If $S_n \sim B(n,p)$ with $q = 1-p$ , then for large $n$ :

\frac{S_n - np}{\sqrt{npq}} \stackrel{d}{\to} N(0,1)

This provides normal approximation to binomial probabilities, useful when $n$ is large.

Worked Examples

Example 1: Applying CLT

Problem:

If $X_1, \ldots, X_{100}$ are i.i.d. with $E[X_i] = 5$ and $\text{Var}(X_i) = 4$ , approximate $P(\sum_{i=1}^{100} X_i > 520)$ .

Solution:

1. By CLT: $\frac{\sum X_i - 500}{\sqrt{100} \times 2} = \frac{\sum X_i - 500}{20} \sim N(0,1)$
2. Transform: $P(\sum X_i > 520) = P(\frac{\sum X_i - 500}{20} > 1)$
3. Standard normal: $P(Z > 1) = 1 - \Phi(1) = 1 - 0.8413 = 0.1587$

Example 2: Normal Approximation to Binomial

Problem:

For $X \sim B(100, 0.3)$ , approximate $P(25 \leq X \leq 35)$ using normal approximation.

Solution:

1. Parameters: $\mu = np = 30$ , $\sigma = \sqrt{npq} = \sqrt{21} \approx 4.58$
2. Standardize: $P(25 \leq X \leq 35) \approx P(\frac{24.5-30}{4.58} \leq Z \leq \frac{35.5-30}{4.58})$
3. Calculate: $P(-1.20 \leq Z \leq 1.20) = 2\Phi(1.20) - 1 = 2(0.8849) - 1 = 0.7698$
4. Note: Used continuity correction (24.5 and 35.5) for better approximation

Theorem Proofs

Rigorous proofs of fundamental limit theorems

Weak Law of Large Numbers

For i.i.d. random variables

X_1, X_2, \ldots, X_n

with

E[X_i] = \mu

and

\text{Var}(X_i) = \sigma^2 < \infty

,

\lim_{n \to \infty} P(|\bar{X}_n - \mu| \geq \epsilon) = 0

for any

\epsilon > 0

Proof (using Chebyshev's Inequality)

1

Compute Moments of Sample Mean

Since $X_i$ are i.i.d.:

E[\bar{X}_n] = E\left[\frac{1}{n}\sum_{i=1}^n X_i\right] = \frac{1}{n}\sum_{i=1}^n E[X_i] = \mu

\text{Var}(\bar{X}_n) = \text{Var}\left(\frac{1}{n}\sum_{i=1}^n X_i\right) = \frac{1}{n^2}\sum_{i=1}^n \text{Var}(X_i) = \frac{\sigma^2}{n}

2

Apply Chebyshev's Inequality

For any $\epsilon > 0$ :

P(|\bar{X}_n - \mu| \geq \epsilon) \leq \frac{\text{Var}(\bar{X}_n)}{\epsilon^2} = \frac{\sigma^2}{n\epsilon^2}

3

Take the Limit

\lim_{n \to \infty} P(|\bar{X}_n - \mu| \geq \epsilon) \leq \lim_{n \to \infty} \frac{\sigma^2}{n\epsilon^2} = 0

Since probabilities are non-negative, this proves convergence in probability. $\blacksquare$

Central Limit Theorem (Sketch)

For i.i.d. random variables

X_1, X_2, \ldots, X_n

with

E[X_i] = \mu

and

0 < \text{Var}(X_i) = \sigma^2 < \infty

,

\frac{\sqrt{n}(\bar{X}_n - \mu)}{\sigma} \stackrel{d}{\to} N(0,1)

Proof Sketch (via Characteristic Functions)

1

Standardize Variables

Let $Z_i = (X_i - \mu)/\sigma$ . Then $E[Z_i] = 0$ , $\text{Var}(Z_i) = 1$ .

Define $S_n^* = \frac{\sum_{i=1}^n Z_i}{\sqrt{n}}$ . We want to show $S_n^* \stackrel{d}{\to} N(0,1)$ .

2

Characteristic Function

Let $\phi(t) = E[e^{itZ_i}]$ be the characteristic function of $Z_i$ . By Taylor expansion:

\phi(t) = 1 + it E[Z_i] - \frac{t^2}{2}E[Z_i^2] + o(t^2) = 1 - \frac{t^2}{2} + o(t^2)

3

Characteristic Function of Sum

Since $Z_i$ are i.i.d., the CF of $S_n^*$ is:

\phi_{S_n^*}(t) = \left[\phi\left(\frac{t}{\sqrt{n}}\right)\right]^n = \left[1 - \frac{t^2}{2n} + o\left(\frac{t^2}{n}\right)\right]^n

4

Take the Limit

\lim_{n \to \infty} \phi_{S_n^*}(t) = \lim_{n \to \infty} \left[1 - \frac{t^2}{2n}\right]^n = e^{-t^2/2}

This is the characteristic function of $N(0,1)$ . By the Lévy continuity theorem, convergence of CFs implies convergence in distribution. $\blacksquare$

Strong Law of Large Numbers (Outline)

For i.i.d. random variables

X_1, X_2, \ldots

with

E[|X_i|] < \infty

and

E[X_i] = \mu

,

P(\lim_{n \to \infty} \bar{X}_n = \mu) = 1

(almost sure convergence)

Key Ideas

1

Truncation

Truncate the variables to reduce them to the bounded case, then show the truncation error vanishes.

2

Borel-Cantelli Lemma

Show that $\sum_{n=1}^\infty P(|\bar{X}_n - \mu| > \epsilon) < \infty$ for any $\epsilon > 0$ .

By Borel-Cantelli, this implies $P(|\bar{X}_n - \mu| > \epsilon \text{ i.o.}) = 0$ , proving almost sure convergence.

3

Fourth Moment Method

For the bounded case with fourth moments, use:

\sum_{n=1}^\infty P(|\bar{X}_n - \mu| > \epsilon) \leq \sum_{n=1}^\infty \frac{E[(\bar{X}_n - \mu)^4]}{\epsilon^4} < \infty

The full proof requires more sophisticated techniques (martingale theory or ergodic theory). $\square$

Practice Quiz

10

Questions

0

Correct

0%

Accuracy

1

What is convergence in distribution?

Not attempted

2

What is the relationship between convergence in probability and convergence in distribution?

Not attempted

3

What does the Weak Law of Large Numbers state?

Not attempted

4

What is the Central Limit Theorem?

Not attempted

5

What is Slutsky's Lemma used for?

Not attempted

6

What is almost sure convergence?

Not attempted

7

What does the Strong Law of Large Numbers state?

Not attempted

8

What is the relationship between convergence types?

Not attempted

9

What is Lévy Continuity Theorem?

Not attempted

10

What is the de Moivre-Laplace Theorem?

Not attempted

Frequently Asked Questions

What is the difference between weak and strong law of large numbers?

The weak law gives convergence in probability of the sample mean to the population mean. The strong law gives almost sure convergence, which is a stronger statement because it holds with probability 1.

When can I use the Central Limit Theorem?

Use the Central Limit Theorem for sums or averages of many independent variables with finite variance. After centering and scaling, the distribution approaches a normal law, which justifies common approximations in inference.

What is the relationship between different types of convergence?

Almost sure convergence implies convergence in probability, and convergence in probability implies convergence in distribution. The reverse implications do not hold in general, except for special cases such as convergence in distribution to a constant.

How do I apply Slutsky's Lemma?

Slutsky's Lemma is used when one sequence converges in distribution and another converges in probability to a constant. It lets you combine them through sums, products, and ratios while preserving convergence in distribution.

What is the practical importance of limit theorems?

Limit theorems justify sample averages, normal approximations, and asymptotic statistical procedures. They are the mathematical backbone of estimation, confidence intervals, and hypothesis testing.

Limit Theorems for Random Variables

Core Concepts

Definition

Lévy Continuity Theorem

Definition

Law of Large Numbers

Theorem:

Kolmogorov SLLN:

Central Limit Theorem

Theorem:

Normal Approximation to Binomial:

Worked Examples

Theorem Proofs

Proof (using Chebyshev's Inequality)

Compute Moments of Sample Mean

Apply Chebyshev's Inequality

Take the Limit

Proof Sketch (via Characteristic Functions)

Standardize Variables

Characteristic Function

Characteristic Function of Sum

Take the Limit

Key Ideas

Truncation

Borel-Cantelli Lemma

Fourth Moment Method

Frequently Asked Questions