Comprehensive mathematical formulas for point estimation theory, evaluation criteria, construction methods, and efficiency bounds
Core definitions and evaluation criteria for point estimators
estimator
θ̂ = θ̂(X₁, X₂, ..., Xₙ)
estimate
θ̂(x₁, x₂, ..., xₙ) (numerical value)
bias
Bias_θ[θ̂] = E_θ[θ̂] - θ
mse
MSE_θ[θ̂] = E_θ[(θ̂ - θ)²] = Var_θ[θ̂] + Bias²_θ[θ̂]
Parameters
θ ∈ Θ (parameter space)
unbiased
E_θ[θ̂] = θ for all θ ∈ Θ
asymptotic unbiased
lim_{n→∞} E_θ[θ̂ₙ] = θ
bias decomposition
E_θ[θ̂] = θ + Bias_θ[θ̂]
bias correction
θ̂* = θ̂ - Bias_θ[θ̂]
Parameters
All θ in parameter space Θ
weak consistency
θ̂ₙ →^P θ (convergence in probability)
strong consistency
P(lim_{n→∞} θ̂ₙ = θ) = 1
mse consistency
lim_{n→∞} MSE_θ[θ̂ₙ] = 0
probability statement
lim_{n→∞} P_θ(|θ̂ₙ - θ| ≥ ε) = 0
Parameters
For all θ ∈ Θ, ε > 0
Mathematical formulations for major parameter estimation approaches
population moment
μₖ = E[X^k] (k-th population moment)
sample moment
aₙ,ₖ = (1/n)∑ᵢ₌₁ⁿ Xᵢᵏ (k-th sample moment)
central moment
νₖ = E[(X - μ)ᵏ] (k-th central moment)
sample central
mₙ,ₖ = (1/n)∑ᵢ₌₁ⁿ (Xᵢ - X̄)ᵏ
likelihood
L(θ; x₁, ..., xₙ) = ∏ᵢ₌₁ⁿ p(xᵢ; θ)
log likelihood
ℓ(θ; x₁, ..., xₙ) = ∑ᵢ₌₁ⁿ log p(xᵢ; θ)
likelihood equation
∂ℓ/∂θⱼ = 0, j = 1, ..., k
mle definition
θ̂ = arg max_θ L(θ; x₁, ..., xₙ)
model
Yᵢ = μᵢ(θ) + εᵢ, i = 1, ..., n
objective
Q(θ) = ∑ᵢ₌₁ⁿ (Yᵢ - μᵢ(θ))²
lse definition
θ̂ = arg min_θ Q(θ)
normal equations
∂Q/∂θⱼ = -2∑ᵢ₌₁ⁿ (Yᵢ - μᵢ(θ))∂μᵢ/∂θⱼ = 0
Information theory and lower bounds for estimator variance
scalar form
I(θ) = E_θ[(∂log p(X;θ)/∂θ)²]
alternative form
I(θ) = -E_θ[∂²log p(X;θ)/∂θ²]
sample information
Iₙ(θ) = nI(θ)
multivariate
I(θ)ⱼₖ = E_θ[∂log p(X;θ)/∂θⱼ · ∂log p(X;θ)/∂θₖ]
scalar bound
Var_θ[ĝ] ≥ [g'(θ)]²/Iₙ(θ)
vector bound
Cov_θ[ĝ] ≥ G(θ)I⁻¹(θ)G^T(θ)
efficiency
eff(ĝ) = [g'(θ)]²/[Iₙ(θ)Var_θ[ĝ]]
asymptotic efficiency
lim_{n→∞} eff(ĝₙ) = 1
Uniformly minimum variance unbiased estimator construction and properties
conditional expectation
ĝ(T) = E[φ(X)|T]
variance reduction
Var_θ[ĝ(T)] ≤ Var_θ[φ(X)]
unbiasedness preservation
E_θ[ĝ(T)] = E_θ[φ(X)] = g(θ)
improvement condition
Equality iff φ(X) = ĝ(T) a.s.
sufficient complete
S complete sufficient statistic
umvue form
ĝ = E[φ(X)|S] (unique UMVUE)
completeness condition
E_θ[h(S)] = 0 ∀θ ⟹ P_θ(h(S) = 0) = 1
uniqueness
ĝ₁ = ĝ₂ a.s. if both UMVUE
Large sample behavior and limiting distributions
mle asymptotic
√n(θ̂ₙ - θ) →^D N(0, I⁻¹(θ))
general form
√n(θ̂ₙ - θ) →^D N(0, Σ(θ))
delta method
√n(g(θ̂ₙ) - g(θ)) →^D N(0, g'(θ)²σ²)
confidence interval
θ̂ₙ ± z_{α/2}/√(nI(θ̂ₙ))
Standard results for frequently encountered distributions
| Distribution | MLE | UMVUE | Fisher Information | CR Bound |
|---|---|---|---|---|
Normal N(μ,σ²) μ ∈ ℝ, σ² > 0 | μ̂ = X̄, σ̂² = Sₙ² | μ̂ = X̄, σ̂² = S² | I(μ) = 1/σ², I(σ²) = 1/(2σ⁴) | Var[μ̂] ≥ σ²/n, Var[σ̂²] ≥ 2σ⁴/n |
Exponential E(λ) λ > 0 | λ̂ = 1/X̄ | λ̂* = (n-1)/(nX̄) | I(λ) = 1/λ² | Var[λ̂] ≥ λ²/n |
Poisson P(λ) λ > 0 | λ̂ = X̄ | λ̂ = X̄ | I(λ) = 1/λ | Var[λ̂] ≥ λ/n |
Bernoulli B(1,p) p ∈ (0,1) | p̂ = X̄ | p̂ = X̄ | I(p) = 1/(p(1-p)) | Var[p̂] ≥ p(1-p)/n |
Deepen your understanding with theory, practice, and applications
Explore distribution families and their estimation properties
Explore Distributions