Learn generative model-based clustering using Gaussian distributions. Master the EM algorithm, understand soft clustering with probability assignments, and handle multi-modal distributions.
Gaussian Mixture Model (GMM) is a generative model that assumes data is generated from a mixture of k Gaussian distributions. Each Gaussian component represents one cluster, and samples are assigned to clusters probabilistically (soft clustering).
Each sample is generated by:
p_M(x) = Σᵢ₌₁ᵏ αᵢ · p(x | μᵢ, Σᵢ)
where Σᵢ αᵢ = 1, αᵢ > 0, and p(x | μᵢ, Σᵢ) is the Gaussian PDF
GMM parameters are estimated using the EM algorithm, which iterates between E-step (expectation) and M-step (maximization):
Calculate the posterior probability that each sample belongs to each Gaussian component:
γⱼᵢ = P(zⱼ = i | xⱼ) = αᵢ · p(xⱼ | μᵢ, Σᵢ) / Σₗ αₗ · p(xⱼ | μₗ, Σₗ)
where zⱼ is the hidden variable indicating which component generated sample xⱼ
Update model parameters to maximize the expected log-likelihood:
αᵢ = (1/m) Σⱼ γⱼᵢ
μᵢ = (Σⱼ γⱼᵢ · xⱼ) / (Σⱼ γⱼᵢ)
Σᵢ = (Σⱼ γⱼᵢ · (xⱼ - μᵢ)(xⱼ - μᵢ)ᵀ) / (Σⱼ γⱼᵢ)
Unlike k-means, GMM provides soft clustering - each sample has a probability distribution over all clusters rather than a hard assignment.
For sample xⱼ, the posterior probabilities γⱼᵢ indicate how likely it belongs to each cluster:
Example probabilities:
This sample is primarily in Cluster 1 but has some membership in other clusters.