Random Variables Limit Theorems

Convergence concepts and asymptotic behavior of random sequences

8-10

Hours Study

Lessons

Advanced

Level

Key Topics

Learning Objectives

Understand different types of convergence for random variable sequences

Master the fundamental law of large numbers and its applications

Apply central limit theorem to practical probability problems

Analyze convergence properties using characteristic functions

Distinguish between weak and strong convergence concepts

Key Topics Overview

Essential concepts in limit theorems for random variables

Convergence in Distribution

Weak convergence of distribution functions

Convergence in Probability

Probability convergence and Slutsky's lemma

Law of Large Numbers

Weak and strong law of large numbers

Central Limit Theorem

Normal approximation for sums of random variables

Convergence in Distribution

Weak convergence and distribution function limits

Definition and Core Concepts

Convergence in distribution describes the limiting behavior of random variable sequences through their distribution functions, focusing on convergence at continuity points.

Distribution Function Weak Convergence:

Let $\{F_n(x)\}$ be a sequence of distribution functions and $F(x)$ be a distribution function. We say $F_n$ converges weakly to $F$ , denoted $F_n \stackrel{w}{\to} F$ , if:

\lim_{n\to\infty}F_n(x)=F(x)

for all continuity points $x$ of $F(x)$ .

Random Variable Convergence in Distribution:

If random variables $\xi_n$ have distribution functions $F_n$ that converge weakly to the distribution function $F$ of random variable $\xi$ , then:

\xi_n \stackrel{d}{\to} \xi

Key Properties and Theorems

Helly's Theorem (First)

Any sequence of distribution functions $\{F_n\}$ contains a subsequence $\{F_{n_k}\}$ that converges weakly to some monotonic, right-continuous function $F$ with $0 \leq F(x) \leq 1$ .

Helly's Theorem (Second)

If $F_n \stackrel{w}{\to} F$ and $g(x)$ is bounded and continuous, then:

\int_{-\infty}^{\infty}g(x)dF_n(x) \to \int_{-\infty}^{\infty}g(x)dF(x)

Lévy Continuity Theorem

$\xi_n \stackrel{d}{\to} \xi$ if and only if their characteristic functions converge: $f_n(t) \to f(t)$ for all $t \in \mathbb{R}$ .

Poisson Approximation

If $\xi_n \sim B(n,p_n)$ and $\lim_{n\to\infty}np_n=\lambda>0$ , then:

\xi_n \stackrel{d}{\to} P(\lambda)

Convergence in Probability

Probability convergence and related theorems

Definition and Properties

Convergence in Probability:

Random variables $\xi_n$ converge in probability to $\xi$ , denoted $\xi_n \stackrel{P}{\to} \xi$ , if for any $\varepsilon > 0$ :

\lim_{n\to\infty}P(|\xi_n-\xi|\geq\varepsilon)=0

Relationship with Convergence in Distribution:

Convergence in probability ⇒ Convergence in distribution
If $\xi_n \stackrel{d}{\to} c$ (constant), then $\xi_n \stackrel{P}{\to} c$
Convergence in distribution does not imply convergence in probability (in general)

Slutsky's Lemma

Slutsky's Lemma provides rules for combining convergent sequences. If $\xi_n \stackrel{d}{\to} \xi$ and $\eta_n \stackrel{P}{\to} c$ (where $c$ is a constant), then:

Addition:

\xi_n+\eta_n \stackrel{d}{\to} \xi+c

Subtraction:

\xi_n-\eta_n \stackrel{d}{\to} \xi-c

Multiplication:

\xi_n\eta_n \stackrel{d}{\to} c\xi

Division:

\xi_n/\eta_n \stackrel{d}{\to} \xi/c \text{ (if } c \neq 0\text{)}

Law of Large Numbers

Weak and strong convergence of sample means

Weak Law of Large Numbers (WLLN)

The weak law describes convergence in probability of sample means to population means.

Theorem	Conditions	Conclusion
Bernoulli WLLN	$\xi_n \sim \text{Bernoulli}(p)$ , i.i.d.	$\frac{1}{n}\sum_{k=1}^n\xi_k \stackrel{P}{\to} p$
Chebyshev WLLN	Independent, $\frac{1}{n^2}\sum_{k=1}^n\text{Var}\xi_k \to 0$	$\frac{1}{n}\sum_{k=1}^n\xi_k - \frac{1}{n}\sum_{k=1}^n E\xi_k \stackrel{P}{\to} 0$
Khintchine WLLN	i.i.d., $E\|\xi_1\|<\infty$ , $E\xi_1=\mu$	$\frac{1}{n}\sum_{k=1}^n\xi_k \stackrel{P}{\to} \mu$

Strong Law of Large Numbers (SLLN)

Almost Sure Convergence:

$\xi_n$ converges almost surely to $\xi$ , denoted $\xi_n \stackrel{a.s.}{\to} \xi$ , if there exists $\Omega_0$ with $P(\Omega_0)=0$ such that for all $\omega \in \Omega\setminus\Omega_0$ :

\lim_{n\to\infty}\xi_n(\omega)=\xi(\omega)

Theorem	Conditions	Conclusion
Borel SLLN	$\xi_n \sim \text{Bernoulli}(p)$ , i.i.d.	$\frac{1}{n}\sum_{k=1}^n\xi_k \stackrel{a.s.}{\to} p$
Kolmogorov SLLN	i.i.d.	$\frac{1}{n}\sum_{k=1}^n\xi_k \stackrel{a.s.}{\to} \mu \iff E\|\xi_1\|<\infty, \mu=E\xi_1$

Convergence Relationships

Almost Sure Convergence

⇓

Convergence in Probability

⇓

Convergence in Distribution

Strong convergence implies weak convergence, but not vice versa

Central Limit Theorem

Normal approximation for sums of random variables

Core Principle

The Central Limit Theorem states that the sum of a large number of independent random variables, when properly normalized, converges in distribution to a normal distribution, regardless of the individual distributions.

Universal Normal Convergence:

\frac{\sum_{k=1}^n \xi_k - E[\sum_{k=1}^n \xi_k]}{\sqrt{\text{Var}[\sum_{k=1}^n \xi_k]}} \stackrel{d}{\to} N(0,1)

Major Central Limit Theorems

de Moivre-Laplace Theorem

Conditions: $\xi_n \sim B(n,p)$ , $q=1-p$

Result: For large $n$ :

\frac{S_n - np}{\sqrt{npq}} \stackrel{d}{\to} N(0,1)

This provides normal approximation to binomial probabilities.

Lindeberg-Lévy Theorem

Conditions: $\{\xi_n\}$ i.i.d., $E\xi_1=a$ , $0<\text{Var}\xi_1=\sigma^2<\infty$

Result:

\frac{\sum_{k=1}^n\xi_k - na}{\sqrt{n}\sigma} \stackrel{d}{\to} N(0,1)

The classical CLT for identical distributions.

Lindeberg-Feller Theorem

Conditions: $\{\xi_n\}$ independent, satisfying Lindeberg condition:

\frac{1}{B_n^2}\sum_{k=1}^n\int_{|x-E\xi_k|\geq\tau B_n}(x-E\xi_k)^2dF_k(x) \to 0

where $B_n^2=\sum_{k=1}^n\text{Var}\xi_k$

Result:

\frac{\sum_{k=1}^n\xi_k - E[\sum_{k=1}^n\xi_k]}{B_n} \stackrel{d}{\to} N(0,1)

Most general form of CLT for non-identical distributions.

Lyapunov Theorem

Conditions: $\{\xi_n\}$ independent, exists $\delta>0$ such that:

\frac{1}{B_n^{2+\delta}}\sum_{k=1}^nE|\xi_k-E\xi_k|^{2+\delta} \to 0

Result: Same as Lindeberg-Feller theorem

Sufficient condition that's easier to verify than Lindeberg condition.

Practical Applications

Binomial Approximation

For $B(n,p)$ with large $n$ :

P(a \leq S_n \leq b) \approx \Phi\left(\frac{b-np}{\sqrt{npq}}\right) - \Phi\left(\frac{a-np}{\sqrt{npq}}\right)

Confidence Intervals

For sample mean $\bar{X}$ with large samples:

\bar{X} \pm z_{\alpha/2} \frac{\sigma}{\sqrt{n}}

provides approximate confidence intervals.

Quality Control

Control charts use CLT to determine if process means have shifted from target values using sample statistics.

Survey Sampling

Polling and survey results rely on CLT to estimate population proportions and construct margin of error bounds.

Supporting Theorems and Inequalities

Important auxiliary results for limit theorems

Borel-Cantelli Lemma

First Lemma: If $\sum_{n=1}^{\infty}P(A_n)<\infty$ , then:

P(\limsup_{n\to\infty}A_n)=0

Second Lemma: If events are independent and $\sum_{n=1}^{\infty}P(A_n)=\infty$ , then:

P(\limsup_{n\to\infty}A_n)=1

Used to prove almost sure convergence results.

Kolmogorov's Inequality

For independent $\{\xi_n\}$ with finite variances:

P\left(\max_{1\leq j\leq n}\left|\sum_{k=1}^j(\xi_k-E\xi_k)\right|\geq\varepsilon\right) \leq \frac{1}{\varepsilon^2}\sum_{k=1}^n\text{Var}\xi_k

Provides bounds on maximum partial sum deviations.

Kolmogorov Three-Series Theorem

For independent $\{\xi_n\}$ , let $\xi_n'=\xi_n I(|\xi_n|\leq c)$ . Then $\sum_{n=1}^{\infty}(\xi_n-E\xi_n)$ converges a.s. iff all three series converge:

\sum_{n=1}^{\infty} P(|\xi_n|>c)<\infty

\sum_{n=1}^{\infty}E\xi_n'

converges

\sum_{n=1}^{\infty}\text{Var}\xi_n'<\infty

Hájek-Rényi Inequality

For independent $\{\xi_n\}$ and positive non-increasing $\{C_n\}$ :

P\left(\max_{m\leq j\leq n}C_j\left|\sum_{k=1}^j(\xi_k-E\xi_k)\right|\geq\varepsilon\right)

\leq \frac{1}{\varepsilon^2}\left(C_m^2\sum_{j=1}^m\text{Var}\xi_j + \sum_{j=m+1}^nC_j^2\text{Var}\xi_j\right)

Generalization of Kolmogorov's inequality.