LA-C10

Available

Course 10: Inner Product Spaces

Inner product spaces add geometric structure to vector spaces, enabling concepts of length, angle, and orthogonality. This course covers inner products, orthonormal bases, projections, least squares approximation, and the fundamental spectral theorem for symmetric matrices.

15-18 hours Advanced Level 10 Objectives

Learning Objectives

Define inner products on real and complex vector spaces.
Understand the Cauchy-Schwarz inequality and its consequences.
Define orthogonality and orthonormal bases.
Master the Gram-Schmidt orthogonalization process.
Compute orthogonal projections onto subspaces.
Apply least squares approximation and solve normal equations.
State and apply the spectral theorem for self-adjoint operators.
Understand orthogonal diagonalization of symmetric matrices.
Connect inner products to geometry (length, angle, distance).
Apply these concepts to function spaces and applications.

Prerequisites

LA-C9: Eigenvalues & Eigenvectors
Vector spaces and bases
Linear independence
Complex numbers (for complex inner products)
Matrix operations

Historical Context

The concept of inner products evolved from the dot product in Euclidean geometry. Augustin-Louis Cauchy (1789–1857) proved the Cauchy-Schwarz inequality in 1821. Jørgen Gram (1850–1916) and Erhard Schmidt (1876–1959) developed the orthogonalization process independently. The spectral theorem, one of the most important results in linear algebra, was proven by David Hilbert (1862–1943) and others in the early 20th century. Inner product spaces, especially Hilbert spaces, are fundamental in quantum mechanics, signal processing, and functional analysis.

1. Inner Products

An inner product is a function that assigns a scalar to each pair of vectors, generalizing the dot product and enabling geometric concepts like length and angle.

Definition 1.1: Inner Product

An inner product on a vector space $V$ over $F$ is a function $\langle \cdot, \cdot \rangle: V \times V \to F$ satisfying:

Linearity: $\langle \alpha u + \beta v, w \rangle = \alpha \langle u, w \rangle + \beta \langle v, w \rangle$
Conjugate symmetry: $\langle v, w \rangle = \overline{\langle w, v \rangle}$ (for real: symmetry)
Positive definiteness: $\langle v, v \rangle \geq 0$ with equality iff $v = 0$

Definition 1.2: Induced Norm

The norm induced by an inner product is $||v|| = \sqrt{\langle v, v \rangle}$ .

Theorem 1.1: Cauchy-Schwarz Inequality

For any vectors $x, y$ in an inner product space:

|\langle x, y \rangle| \leq ||x|| \cdot ||y||

Equality holds if and only if $x$ and $y$ are linearly dependent.

Example 1.1: Standard Inner Products

$\mathbb{R}^n$ : $\langle x, y \rangle = x^T y = \sum x_i y_i$
$\mathbb{C}^n$ : $\langle x, y \rangle = x^* y = \sum \bar{x}_i y_i$
$L^2[a,b]$ : $\langle f, g \rangle = \int_a^b f(t) \overline{g(t)} dt$

2. Orthogonality

Orthogonality generalizes perpendicularity. Orthonormal bases provide the most convenient coordinate systems for computation.

Definition 2.1: Orthogonal and Orthonormal

Vectors $u, v$ are orthogonal if $\langle u, v \rangle = 0$ .

A set $\{e_1, \ldots, e_k\}$ is orthonormal if $\langle e_i, e_j \rangle = \delta_{ij}$ (Kronecker delta).

Theorem 2.1: Independence of Orthogonal Vectors

An orthogonal set of nonzero vectors is linearly independent.

Theorem 2.2: Fourier Coefficients

If $\{e_1, \ldots, e_n\}$ is an orthonormal basis, then for any $v = \sum c_i e_i$ :

c_i = \langle v, e_i \rangle

These are called Fourier coefficients.

Theorem 2.3: Parseval's Identity

For an orthonormal basis, $||v||^2 = \sum |c_i|^2$ where $c_i = \langle v, e_i \rangle$ .

3. Gram-Schmidt Process

The Gram-Schmidt process converts any basis into an orthonormal basis, proving that every finite-dimensional inner product space has an orthonormal basis.

Definition 3.1: Gram-Schmidt Algorithm

Given linearly independent vectors $\{v_1, \ldots, v_n\}$ :

$e_1 = v_1 / ||v_1||$ (normalize first vector)
For $k = 2, \ldots, n$ :

$u_k = v_k - \sum_{i=1}^{k-1} \langle v_k, e_i \rangle e_i$ (subtract projections)
$e_k = u_k / ||u_k||$ (normalize)

Result: $\{e_1, \ldots, e_n\}$ is orthonormal with $\text{span}\{e_1, \ldots, e_k\} = \text{span}\{v_1, \ldots, v_k\}$ .

Theorem 3.1: QR Decomposition

If $A$ has linearly independent columns, then $A = QR$ where $Q$ has orthonormal columns (from Gram-Schmidt) and $R$ is upper triangular.

Example 3.1: Gram-Schmidt Example

For $v_1 = (1,1,0), v_2 = (1,0,1)$ in $\mathbb{R}^3$ :

$e_1 = (1/\sqrt{2}, 1/\sqrt{2}, 0)$
$u_2 = (1,0,1) - \langle (1,0,1), e_1 \rangle e_1 = (1/2, -1/2, 1)$
$e_2 = u_2 / ||u_2||$

4. Orthogonal Projections

The orthogonal projection of a vector onto a subspace is the closest point in that subspace. This is fundamental for least squares approximation.

Definition 4.1: Orthogonal Projection

The orthogonal projection of $v$ onto subspace $U$ is the unique vector $\text{proj}_U(v) \in U$ such that $v - \text{proj}_U(v) \in U^\perp$ .

Theorem 4.1: Projection Formula

If $\{e_1, \ldots, e_k\}$ is an orthonormal basis for $U$ , then:

\text{proj}_U(v) = \sum_{i=1}^k \langle v, e_i \rangle e_i

Theorem 4.2: Best Approximation

$\text{proj}_U(v)$ minimizes $||v - u||$ over all $u \in U$ .

Definition 4.2: Least Squares

For $Ax \approx b$ (overdetermined), the least squares solution minimizes $||Ax - b||$ .

Theorem 4.3: Normal Equation

The least squares solution satisfies $A^T A x = A^T b$ (the normal equation).

Example 4.1: Linear Regression

For data points $(x_i, y_i)$ , fitting $y = ax + b$ minimizes $\sum (y_i - ax_i - b)^2$ . This is least squares with $A = [x \, 1], b = y$ .

5. Spectral Theorem

The spectral theorem is one of the most important results in linear algebra, characterizing self-adjoint operators and enabling orthogonal diagonalization of symmetric matrices.

Definition 5.1: Self-Adjoint Operator

An operator $T$ on an inner product space is self-adjoint if $T = T^*$ , where $T^*$ is the adjoint satisfying $\langle Tx, y \rangle = \langle x, T^* y \rangle$ .

For real matrices with standard inner product, self-adjoint means symmetric ( $A = A^T$ ).

Theorem 5.1: Eigenvalues of Self-Adjoint Operators

All eigenvalues of a self-adjoint operator are real.

Theorem 5.2: Orthogonality of Eigenvectors

Eigenvectors corresponding to distinct eigenvalues of a self-adjoint operator are orthogonal.

Theorem 5.3: Spectral Theorem

Every self-adjoint operator on a finite-dimensional inner product space has an orthonormal basis of eigenvectors. Equivalently, every real symmetric matrix is orthogonally diagonalizable: $A = QDQ^T$ where $Q$ is orthogonal and $D$ is diagonal with real entries.

Corollary 5.1: Spectral Decomposition

If $A = QDQ^T$ with eigenvalues $\lambda_i$ and orthonormal eigenvectors $q_i$ , then:

A = \sum_{i=1}^n \lambda_i q_i q_i^T

This is the spectral decomposition.

Remark 5.1: Applications

The spectral theorem is fundamental in quantum mechanics (observables are self-adjoint), principal component analysis, and many areas of applied mathematics.

Course 10 Practice Quiz

Questions

Correct

Accuracy

An inner product

\langle \cdot, \cdot \rangle

must satisfy:

Easy

Not attempted

Cauchy-Schwarz states that:

Medium

Not attempted

Vectors

u, v

are orthogonal if:

Easy

Not attempted

\{e_1,...,e_k\}

is orthonormal basis for

U

, then

\text{proj}_U(v) =

Medium

Not attempted

The normal equation for least squares

Ax \approx b

is:

Medium

Not attempted

A real symmetric matrix has:

Easy

Not attempted

The spectral theorem says self-adjoint

T

has:

Medium

Not attempted

Gram-Schmidt takes as input:

Easy

Not attempted

For orthonormal basis

\{e_1,...,e_n\}

, coefficient

c_k

v = \sum c_i e_i

is:

Medium

Not attempted

Orthogonal diagonalization means

A = QDQ^T

where:

Medium

Not attempted

Frequently Asked Questions

What's the geometric meaning of inner products?

Inner products measure 'angle' and 'length'. ||v|| = √⟨v,v⟩ gives length. cos θ = ⟨x,y⟩/(||x||·||y||) defines angle. Orthogonality (⟨x,y⟩ = 0) means perpendicular.

Why are orthonormal bases so useful?

Three main reasons: (1) Coefficients are trivial to compute as inner products, no matrix inversion needed. (2) The Gram matrix is the identity, simplifying all computations. (3) Parseval's identity gives ||v||² = Σ|cᵢ|², preserving norms.

How does Gram-Schmidt work?

Start with first vector, normalize it. For each subsequent vector, subtract its projections onto all previous orthonormal vectors, then normalize. This makes each new vector orthogonal to all previous ones.

What is least squares approximation?

When Ax = b has no solution, find x that minimizes ||Ax - b||. This is equivalent to finding the projection of b onto col(A). The solution satisfies the normal equation AᵀAx = Aᵀb.

What does the spectral theorem say?

Every self-adjoint (symmetric/Hermitian) operator has an orthonormal basis of eigenvectors with real eigenvalues. This means A = QDQᵀ where Q is orthogonal and D is diagonal with real entries.