MathIsimple – Simple, Friendly Math Tools & Learning

Core Concepts

Convergence in Distribution

Definition

Distribution functions $F_n$ converge weakly to $F$ , denoted $F_n \stackrel{w}{\to} F$ , if:

\lim_{n\to\infty}F_n(x)=F(x)

for all continuity points $x$ of $F(x)$ .

Lévy Continuity Theorem

\xi_n \stackrel{d}{\to} \xi \iff f_n(t) \to f(t) \text{ for all } t \in \mathbb{R}

Convergence in distribution is equivalent to convergence of characteristic functions.

Convergence in Probability

Definition

\xi_n \stackrel{P}{\to} \xi \iff \lim_{n\to\infty}P(|\xi_n-\xi|\geq\varepsilon)=0 \text{ for all } \varepsilon > 0

Convergence in probability is stronger than convergence in distribution.

Slutsky's Lemma: If $\xi_n \stackrel{d}{\to} \xi$ and $\eta_n \stackrel{P}{\to} c$ , then $\xi_n + \eta_n \stackrel{d}{\to} \xi + c$ , $\xi_n\eta_n \stackrel{d}{\to} c\xi$ , etc.

Law of Large Numbers

Weak Law of Large Numbers (WLLN)

Theorem:

\text{If } \{\xi_n\} \text{ i.i.d. with } E|\xi_1| < \infty, \text{ then } \frac{1}{n}\sum_{k=1}^n \xi_k \stackrel{P}{\to} E\xi_1

The sample mean converges in probability to the population mean.

Strong Law of Large Numbers (SLLN)

Kolmogorov SLLN:

\text{If } \{\xi_n\} \text{ i.i.d., then } \frac{1}{n}\sum_{k=1}^n \xi_k \stackrel{a.s.}{\to} E\xi_1 \iff E|\xi_1| < \infty

The sample mean converges almost surely to the population mean if and only if the expectation is finite.

Central Limit Theorem

Lindeberg-Lévy CLT

Theorem:

If $\{\xi_n\}$ are i.i.d. with $E\xi_1 = \mu$ and $0 < \text{Var}(\xi_1) = \sigma^2 < \infty$ , then:

\frac{\sum_{k=1}^n \xi_k - n\mu}{\sqrt{n}\sigma} \stackrel{d}{\to} N(0,1)

The normalized sum converges in distribution to standard normal distribution.

de Moivre-Laplace Theorem

Normal Approximation to Binomial:

If $S_n \sim B(n,p)$ with $q = 1-p$ , then for large $n$ :

\frac{S_n - np}{\sqrt{npq}} \stackrel{d}{\to} N(0,1)

This provides normal approximation to binomial probabilities, useful when $n$ is large.

Worked Examples

Example 1: Applying CLT

Problem:

If $X_1, \ldots, X_{100}$ are i.i.d. with $E[X_i] = 5$ and $\text{Var}(X_i) = 4$ , approximate $P(\sum_{i=1}^{100} X_i > 520)$ .

Solution:

1. By CLT: $\frac{\sum X_i - 500}{\sqrt{100} \times 2} = \frac{\sum X_i - 500}{20} \sim N(0,1)$
2. Transform: $P(\sum X_i > 520) = P(\frac{\sum X_i - 500}{20} > 1)$
3. Standard normal: $P(Z > 1) = 1 - \Phi(1) = 1 - 0.8413 = 0.1587$

Example 2: Normal Approximation to Binomial

Problem:

For $X \sim B(100, 0.3)$ , approximate $P(25 \leq X \leq 35)$ using normal approximation.

Solution:

1. Parameters: $\mu = np = 30$ , $\sigma = \sqrt{npq} = \sqrt{21} \approx 4.58$
2. Standardize: $P(25 \leq X \leq 35) \approx P(\frac{24.5-30}{4.58} \leq Z \leq \frac{35.5-30}{4.58})$
3. Calculate: $P(-1.20 \leq Z \leq 1.20) = 2\Phi(1.20) - 1 = 2(0.8849) - 1 = 0.7698$
4. Note: Used continuity correction (24.5 and 35.5) for better approximation

Theorem Proofs

Rigorous proofs of fundamental limit theorems

Weak Law of Large Numbers

For i.i.d. random variables

X_1, X_2, \ldots, X_n

with

E[X_i] = \mu

and

\text{Var}(X_i) = \sigma^2 < \infty

,

\lim_{n \to \infty} P(|\bar{X}_n - \mu| \geq \epsilon) = 0

for any

\epsilon > 0

Proof (using Chebyshev's Inequality)

1

Compute Moments of Sample Mean

Since $X_i$ are i.i.d.:

E[\bar{X}_n] = E\left[\frac{1}{n}\sum_{i=1}^n X_i\right] = \frac{1}{n}\sum_{i=1}^n E[X_i] = \mu

\text{Var}(\bar{X}_n) = \text{Var}\left(\frac{1}{n}\sum_{i=1}^n X_i\right) = \frac{1}{n^2}\sum_{i=1}^n \text{Var}(X_i) = \frac{\sigma^2}{n}

2

Apply Chebyshev's Inequality

For any $\epsilon > 0$ :

P(|\bar{X}_n - \mu| \geq \epsilon) \leq \frac{\text{Var}(\bar{X}_n)}{\epsilon^2} = \frac{\sigma^2}{n\epsilon^2}

3

Take the Limit

\lim_{n \to \infty} P(|\bar{X}_n - \mu| \geq \epsilon) \leq \lim_{n \to \infty} \frac{\sigma^2}{n\epsilon^2} = 0

Since probabilities are non-negative, this proves convergence in probability. $\blacksquare$

Central Limit Theorem (Sketch)

For i.i.d. random variables

X_1, X_2, \ldots, X_n

with

E[X_i] = \mu

and

0 < \text{Var}(X_i) = \sigma^2 < \infty

,

\frac{\sqrt{n}(\bar{X}_n - \mu)}{\sigma} \stackrel{d}{\to} N(0,1)

Proof Sketch (via Characteristic Functions)

1

Standardize Variables

Let $Z_i = (X_i - \mu)/\sigma$ . Then $E[Z_i] = 0$ , $\text{Var}(Z_i) = 1$ .

Define $S_n^* = \frac{\sum_{i=1}^n Z_i}{\sqrt{n}}$ . We want to show $S_n^* \stackrel{d}{\to} N(0,1)$ .

2

Characteristic Function

Let $\phi(t) = E[e^{itZ_i}]$ be the characteristic function of $Z_i$ . By Taylor expansion:

\phi(t) = 1 + it E[Z_i] - \frac{t^2}{2}E[Z_i^2] + o(t^2) = 1 - \frac{t^2}{2} + o(t^2)

3

Characteristic Function of Sum

Since $Z_i$ are i.i.d., the CF of $S_n^*$ is:

\phi_{S_n^*}(t) = \left[\phi\left(\frac{t}{\sqrt{n}}\right)\right]^n = \left[1 - \frac{t^2}{2n} + o\left(\frac{t^2}{n}\right)\right]^n

4

Take the Limit

\lim_{n \to \infty} \phi_{S_n^*}(t) = \lim_{n \to \infty} \left[1 - \frac{t^2}{2n}\right]^n = e^{-t^2/2}

This is the characteristic function of $N(0,1)$ . By the Lévy continuity theorem, convergence of CFs implies convergence in distribution. $\blacksquare$

Strong Law of Large Numbers (Outline)

For i.i.d. random variables

X_1, X_2, \ldots

with

E[|X_i|] < \infty

and

E[X_i] = \mu

,

P(\lim_{n \to \infty} \bar{X}_n = \mu) = 1

(almost sure convergence)

Key Ideas

1

Truncation

Truncate the variables to reduce them to the bounded case, then show the truncation error vanishes.

2

Borel-Cantelli Lemma

Show that $\sum_{n=1}^\infty P(|\bar{X}_n - \mu| > \epsilon) < \infty$ for any $\epsilon > 0$ .

By Borel-Cantelli, this implies $P(|\bar{X}_n - \mu| > \epsilon \text{ i.o.}) = 0$ , proving almost sure convergence.

3

Fourth Moment Method

For the bounded case with fourth moments, use:

\sum_{n=1}^\infty P(|\bar{X}_n - \mu| > \epsilon) \leq \sum_{n=1}^\infty \frac{E[(\bar{X}_n - \mu)^4]}{\epsilon^4} < \infty

The full proof requires more sophisticated techniques (martingale theory or ergodic theory). $\square$

Practice Quiz

10

Questions

0

Correct

0%

Accuracy

1

What is convergence in distribution?

Not attempted

2

What is the relationship between convergence in probability and convergence in distribution?

Not attempted

3

What does the Weak Law of Large Numbers state?

Not attempted

4

What is the Central Limit Theorem?

Not attempted

5

What is Slutsky's Lemma used for?

Not attempted

6

What is almost sure convergence?

Not attempted

7

What does the Strong Law of Large Numbers state?

Not attempted

8

What is the relationship between convergence types?

Not attempted

9

What is Lévy Continuity Theorem?

Not attempted

10

What is the de Moivre-Laplace Theorem?

Not attempted

Frequently Asked Questions

What is the difference between weak and strong law of large numbers?

Weak Law states convergence in probability: sample mean converges to population mean with probability approaching 1. Strong Law states almost sure convergence: sample mean converges to population mean with probability 1. Strong Law is stronger and requires finite expectation.

When can I use the Central Limit Theorem?

CLT applies when you have a sum of many independent random variables with finite variance. The normalized sum (subtract mean, divide by standard deviation) converges to standard normal. This allows normal approximations for sums, averages, and sample statistics.

What is the relationship between different types of convergence?

Almost sure convergence is strongest, implying convergence in probability, which in turn implies convergence in distribution. The reverse implications are not generally true. However, convergence in distribution to a constant implies convergence in probability to that constant.

How do I apply Slutsky's Lemma?

Use Slutsky's Lemma when you have one sequence converging in distribution and another converging in probability to a constant. You can then combine them: sums, products, and ratios all converge in distribution to the expected limits. This is essential for asymptotic statistics.

What is the practical importance of limit theorems?

Limit theorems justify statistical methods: LLN justifies using sample means as estimates, CLT enables normal approximations for confidence intervals and hypothesis tests, and convergence theory provides theoretical foundation for asymptotic inference. They are fundamental to all statistical practice.

Limit Theorems for Random Variables

Core Concepts

Definition

Lévy Continuity Theorem

Definition

Law of Large Numbers

Theorem:

Kolmogorov SLLN:

Central Limit Theorem

Theorem:

Normal Approximation to Binomial:

Worked Examples

Theorem Proofs

Proof (using Chebyshev's Inequality)

Compute Moments of Sample Mean

Apply Chebyshev's Inequality

Take the Limit

Proof Sketch (via Characteristic Functions)

Standardize Variables

Characteristic Function

Characteristic Function of Sum

Take the Limit

Key Ideas

Truncation

Borel-Cantelli Lemma

Fourth Moment Method

Frequently Asked Questions