Orthogonality | Inner Product Spaces | Linear Algebra

1. Orthogonal and Orthonormal Sets

Definition 7.9: Orthogonal Set

A set of vectors $\{v_1, v_2, \ldots, v_k\}$ is orthogonal if every pair of distinct vectors is orthogonal:

\langle v_i, v_j \rangle = 0 \quad \text{for all } i \neq j

Definition 7.10: Orthonormal Set

A set is orthonormal if it is orthogonal and each vector has unit norm:

\langle e_i, e_j \rangle = \delta_{ij} = \begin{cases} 1 & \text{if } i = j \\ 0 & \text{if } i \neq j \end{cases}

Remark 7.10: Kronecker Delta

The symbol $\delta_{ij}$ is the Kronecker delta, equal to 1 when $i = j$ and 0 otherwise. It compactly expresses the orthonormality condition.

Example 7.18: Standard Basis is Orthonormal

In $\mathbb{R}^3$ , the standard basis $\{e_1, e_2, e_3\}$ where:

e_1 = (1,0,0)^T, \quad e_2 = (0,1,0)^T, \quad e_3 = (0,0,1)^T

is orthonormal: $\langle e_i, e_j \rangle = \delta_{ij}$ .

Example 7.19: Orthogonal but Not Orthonormal

The set $\{(2,0)^T, (0,3)^T\}$ is orthogonal (inner product is 0) but not orthonormal (vectors don't have unit length).

Example 7.20: Normalizing an Orthogonal Set

To make $\{(2,0)^T, (0,3)^T\}$ orthonormal, normalize each vector:

e_1 = \frac{1}{2}(2,0)^T = (1,0)^T, \quad e_2 = \frac{1}{3}(0,3)^T = (0,1)^T

Theorem 7.15: Orthogonal Sets are Linearly Independent

An orthogonal set of nonzero vectors is linearly independent.

Proof:

Suppose $c_1 v_1 + c_2 v_2 + \cdots + c_k v_k = 0$ for orthogonal nonzero $v_i$ .

Take inner product with $v_j$ :

\langle c_1 v_1 + \cdots + c_k v_k, v_j \rangle = c_j \langle v_j, v_j \rangle = c_j \|v_j\|^2 = 0

Since $v_j \neq 0$ , we have $\|v_j\|^2 > 0$ , so $c_j = 0$ . This holds for all $j$ .

∎

Corollary 7.3: Size Bound

An orthogonal set of nonzero vectors in an $n$ -dimensional space has at most $n$ vectors.

Example 7.20a: Orthogonal Set in ℝ³

The set $\{(1, 1, 0)^T, (1, -1, 0)^T, (0, 0, 2)^T\}$ is orthogonal:

$\langle (1,1,0), (1,-1,0) \rangle = 1 - 1 + 0 = 0$ ✓
$\langle (1,1,0), (0,0,2) \rangle = 0 + 0 + 0 = 0$ ✓
$\langle (1,-1,0), (0,0,2) \rangle = 0 + 0 + 0 = 0$ ✓

To make orthonormal, normalize: $\{\frac{1}{\sqrt{2}}(1,1,0)^T, \frac{1}{\sqrt{2}}(1,-1,0)^T, (0,0,1)^T\}$

Theorem 7.15a: Orthogonality and Components

If $\{v_1, \ldots, v_k\}$ is orthogonal and $v = \sum_{i=1}^k c_i v_i$ , then:

c_j = \frac{\langle v, v_j \rangle}{\|v_j\|^2}

Proof:

Taking inner product of $v$ with $v_j$ :

\langle v, v_j \rangle = \left\langle \sum_i c_i v_i, v_j \right\rangle = \sum_i c_i \langle v_i, v_j \rangle = c_j \|v_j\|^2

since $\langle v_i, v_j \rangle = 0$ for $i \neq j$ .

∎

Remark 7.10a: Simplification with Orthonormal Sets

For orthonormal sets, the denominator $\|v_j\|^2 = 1$ , so $c_j = \langle v, e_j \rangle$ . This is why orthonormal sets are preferred for computation.

Example 7.20b: Checking Orthogonality in Function Spaces

On $C[-1, 1]$ , show $f(x) = 1$ and $g(x) = x$ are orthogonal:

\langle f, g \rangle = \int_{-1}^{1} 1 \cdot x\, dx = \left[\frac{x^2}{2}\right]_{-1}^{1} = \frac{1}{2} - \frac{1}{2} = 0

But $f(x) = 1$ and $h(x) = x^2$ are NOT orthogonal:

\langle f, h \rangle = \int_{-1}^{1} x^2\, dx = \frac{2}{3} \neq 0

Theorem 7.15b: Pythagorean Theorem for Orthogonal Sets

If $v_1, \ldots, v_k$ are pairwise orthogonal:

\|v_1 + v_2 + \cdots + v_k\|^2 = \|v_1\|^2 + \|v_2\|^2 + \cdots + \|v_k\|^2

Proof:

Expand the norm squared:

\|\sum_i v_i\|^2 = \langle \sum_i v_i, \sum_j v_j \rangle = \sum_{i,j} \langle v_i, v_j \rangle = \sum_i \|v_i\|^2

since $\langle v_i, v_j \rangle = 0$ for $i \neq j$ .

∎

Example 7.20c: Pythagorean Theorem Application

For orthogonal vectors $v_1 = (3, 0)^T$ and $v_2 = (0, 4)^T$ :

\|v_1 + v_2\|^2 = \|(3, 4)\|^2 = 25 = 9 + 16 = \|v_1\|^2 + \|v_2\|^2

This is the classic 3-4-5 right triangle!

Definition 7.10a: Orthogonal Matrix

A square matrix $Q$ is orthogonal if its columns form an orthonormal set, equivalently:

Q^T Q = I \iff Q^{-1} = Q^T

For complex matrices, the analogous concept is unitary: $U^H U = I$ .

Example 7.20d: Rotation Matrix is Orthogonal

The 2D rotation matrix is orthogonal:

R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}

Verify: $R^T R = \begin{pmatrix} \cos^2\theta + \sin^2\theta & 0 \\ 0 & \cos^2\theta + \sin^2\theta \end{pmatrix} = I$

Remark 7.10b: Properties of Orthogonal Matrices

Orthogonal matrices preserve:

Lengths: $\|Qx\| = \|x\|$
Angles: $\langle Qx, Qy \rangle = \langle x, y \rangle$
Distances: $\|Qx - Qy\| = \|x - y\|$

They represent rotations and reflections—rigid motions of space.

Theorem 7.15c: Determinant of Orthogonal Matrix

If $Q$ is orthogonal, then $\det(Q) = \pm 1$ :

\det(Q^T Q) = \det(I) = 1 \implies (\det Q)^2 = 1

If $\det Q = 1$ , $Q$ is a rotation. If $\det Q = -1$ , $Q$ includes a reflection.

Example 7.20e: Reflection Matrix

The reflection across the line $y = x$ in $\mathbb{R}^2$ :

R = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}

Verify: $R^T R = I$ , $\det R = -1$ (reflection, not rotation).

Definition 7.10b: Unitary Matrix

A complex matrix $U$ is unitary if $U^H U = I$ where $U^H = \bar{U}^T$ .

Unitary matrices are the complex analog of orthogonal matrices.

Example 7.20f: Unitary Matrix

The matrix $U = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ i & -i \end{pmatrix}$ is unitary:

U^H U = \frac{1}{2}\begin{pmatrix} 1 & -i \\ 1 & i \end{pmatrix}\begin{pmatrix} 1 & 1 \\ i & -i \end{pmatrix} = I

Remark 7.10c: Special Orthogonal Group

The set of $n \times n$ orthogonal matrices forms a group $O(n)$ . The subgroup with $\det = 1$ is $SO(n)$ (special orthogonal group, pure rotations).

2. Orthonormal Bases

Definition 7.11: Orthonormal Basis

An orthonormal basis is an orthonormal set that is also a basis (spans the space).

Theorem 7.16: Existence of Orthonormal Bases

Every finite-dimensional inner product space has an orthonormal basis.

Remark 7.11: Construction

The Gram-Schmidt process (next module) provides an explicit algorithm to construct orthonormal bases from any basis.

Example 7.21: Orthonormal Basis for ℝ²

The set $\left\{ \frac{1}{\sqrt{2}}(1,1)^T, \frac{1}{\sqrt{2}}(1,-1)^T \right\}$ is an orthonormal basis for $\mathbb{R}^2$ :

Orthogonal: $\frac{1}{2}(1 \cdot 1 + 1 \cdot (-1)) = 0$ ✓
Unit length: $\frac{1}{2}(1 + 1) = 1$ ✓ for both
Spans $\mathbb{R}^2$ : 2 linearly independent vectors in 2D ✓

Theorem 7.17: Fourier Coefficients

Let $\{e_1, \ldots, e_n\}$ be an orthonormal basis. For any $v$ :

v = \sum_{i=1}^{n} c_i e_i \quad \text{where} \quad c_i = \langle v, e_i \rangle

Proof:

Since $\{e_i\}$ is a basis, $v = \sum c_i e_i$ for unique scalars $c_i$ .

Taking inner product with $e_j$ :

\langle v, e_j \rangle = \left\langle \sum_{i=1}^n c_i e_i, e_j \right\rangle = \sum_{i=1}^n c_i \langle e_i, e_j \rangle = \sum_{i=1}^n c_i \delta_{ij} = c_j

∎

Remark 7.12: Computational Advantage

With an orthonormal basis, finding coefficients is trivial—just compute inner products! For a general basis, you'd need to solve a linear system (invert a matrix).

Example 7.22: Computing Fourier Coefficients

Express $v = (3, 4)^T$ in the orthonormal basis $\{e_1, e_2\}$ where $e_1 = \frac{1}{\sqrt{2}}(1,1)^T$ , $e_2 = \frac{1}{\sqrt{2}}(1,-1)^T$ :

c_1 = \langle v, e_1 \rangle = \frac{1}{\sqrt{2}}(3 + 4) = \frac{7}{\sqrt{2}}

c_2 = \langle v, e_2 \rangle = \frac{1}{\sqrt{2}}(3 - 4) = \frac{-1}{\sqrt{2}}

Verify: $\frac{7}{\sqrt{2}} \cdot \frac{1}{\sqrt{2}}(1,1)^T + \frac{-1}{\sqrt{2}} \cdot \frac{1}{\sqrt{2}}(1,-1)^T = \frac{7}{2}(1,1)^T - \frac{1}{2}(1,-1)^T = (3,4)^T$ ✓

Example 7.22a: Orthonormal Basis in ℝ³

The set $\{e_1, e_2, e_3\}$ is orthonormal in $\mathbb{R}^3$ :

e_1 = \frac{1}{\sqrt{3}}(1,1,1)^T, \quad e_2 = \frac{1}{\sqrt{2}}(1,-1,0)^T, \quad e_3 = \frac{1}{\sqrt{6}}(1,1,-2)^T

Express $v = (1, 2, 3)^T$ :

c_1 = \langle v, e_1 \rangle = \frac{1}{\sqrt{3}}(1+2+3) = \frac{6}{\sqrt{3}} = 2\sqrt{3}

c_2 = \langle v, e_2 \rangle = \frac{1}{\sqrt{2}}(1-2) = -\frac{1}{\sqrt{2}}

c_3 = \langle v, e_3 \rangle = \frac{1}{\sqrt{6}}(1+2-6) = -\frac{3}{\sqrt{6}}

Theorem 7.17a: Gram Matrix of Orthonormal Basis

If $\{e_1, \ldots, e_n\}$ is orthonormal, its Gram matrix is the identity:

G_{ij} = \langle e_i, e_j \rangle = \delta_{ij} \implies G = I

Remark 7.12a: Coordinate Transformations

The matrix $P$ whose columns are orthonormal basis vectors satisfies $P^T P = I$ . To change from standard coordinates to orthonormal coordinates:

[v]_{\text{new}} = P^T [v]_{\text{standard}}

Example 7.22b: Coordinate Change

With orthonormal basis $P = \begin{pmatrix} 1/\sqrt{2} & 1/\sqrt{2} \\ 1/\sqrt{2} & -1/\sqrt{2} \end{pmatrix}$ , transform $v = (3, 4)^T$ :

[v]_{\text{new}} = P^T v = \begin{pmatrix} 1/\sqrt{2} & 1/\sqrt{2} \\ 1/\sqrt{2} & -1/\sqrt{2} \end{pmatrix} \begin{pmatrix} 3 \\ 4 \end{pmatrix} = \begin{pmatrix} 7/\sqrt{2} \\ -1/\sqrt{2} \end{pmatrix}

Theorem 7.17b: Matrix Representation in Orthonormal Basis

If $\{e_i\}$ is orthonormal and $T$ is a linear operator, the matrix representation has:

A_{ij} = \langle Te_j, e_i \rangle

Remark 7.12b: Self-Adjoint Operators

In orthonormal coordinates, the adjoint $T^*$ has matrix $A^*$ (conjugate transpose). $T$ is self-adjoint iff $A = A^*$ (Hermitian).

Example 7.22c: Change of Basis Example

Transform from standard basis to orthonormal basis $\{u_1, u_2\}$ where $u_1 = \frac{1}{\sqrt{5}}(2, 1)^T$ , $u_2 = \frac{1}{\sqrt{5}}(1, -2)^T$ :

P = \begin{pmatrix} 2/\sqrt{5} & 1/\sqrt{5} \\ 1/\sqrt{5} & -2/\sqrt{5} \end{pmatrix}

For $v = (3, 1)^T$ : $[v]_{\text{new}} = P^T v = \frac{1}{\sqrt{5}}(7, 1)^T$

Theorem 7.17c: Uniqueness of Orthonormal Representation

Every vector $v$ has a unique representation in any orthonormal basis. The coefficients $c_i = \langle v, e_i \rangle$ are determined entirely by $v$ and the basis.

Remark 7.12c: Comparison: Orthonormal vs General Basis

Property	General Basis	Orthonormal Basis
Find coefficients	Solve linear system	Inner products
Gram matrix	G (positive definite)	I (identity)
Norm formula	√(cᵀGc)	√(Σ\|cᵢ\|²)
Inner product	aᵀGb	Σaᵢb̄ᵢ

3. Parseval's Identity and Bessel's Inequality

Theorem 7.18: Parseval's Identity

For orthonormal basis $\{e_1, \ldots, e_n\}$ and $v = \sum c_i e_i$ :

\|v\|^2 = \sum_{i=1}^{n} |c_i|^2 = \sum_{i=1}^{n} |\langle v, e_i \rangle|^2

Proof:

\|v\|^2 = \langle v, v \rangle = \left\langle \sum_i c_i e_i, \sum_j c_j e_j \right\rangle = \sum_{i,j} c_i \bar{c_j} \langle e_i, e_j \rangle = \sum_i |c_i|^2

∎

Remark 7.13: Norm Preservation

Parseval's identity says the norm of a vector equals the norm of its coefficient vector. Orthonormal bases preserve distances!

Theorem 7.19: Bessel's Inequality

For any orthonormal set $\{e_1, \ldots, e_k\}$ (not necessarily a basis) and any vector $v$ :

\sum_{i=1}^{k} |\langle v, e_i \rangle|^2 \leq \|v\|^2

Proof:

Let $w = v - \sum_{i=1}^k \langle v, e_i \rangle e_i$ (residual after projecting onto span{eᵢ}).

Then $\langle w, e_j \rangle = 0$ for all $j$ , so $w$ is orthogonal to each $e_i$ .

\|v\|^2 = \|w\|^2 + \sum_i |\langle v, e_i \rangle|^2 \geq \sum_i |\langle v, e_i \rangle|^2

∎

Corollary 7.4: Bessel Becomes Parseval

Bessel's inequality becomes Parseval's equality iff $\{e_i\}$ is a complete orthonormal basis.

Example 7.23: Bessel in Action

In $\mathbb{R}^3$ , let $e_1 = (1,0,0)^T, e_2 = (0,1,0)^T$ (not a basis, just 2 vectors).

For $v = (1,2,3)^T$ :

\langle v, e_1 \rangle = 1, \quad \langle v, e_2 \rangle = 2

Bessel: $1^2 + 2^2 = 5 \leq 1 + 4 + 9 = 14 = \|v\|^2$ ✓

Example 7.23a: Parseval Verification

For $v = (1, 2, 3)^T$ in $\mathbb{R}^3$ with standard orthonormal basis:

\|v\|^2 = 1 + 4 + 9 = 14

\sum |c_i|^2 = |\langle v, e_1 \rangle|^2 + |\langle v, e_2 \rangle|^2 + |\langle v, e_3 \rangle|^2 = 1 + 4 + 9 = 14

Parseval holds: $\|v\|^2 = \sum |c_i|^2$ ✓

Theorem 7.19a: Polarization with Fourier Coefficients

For orthonormal basis $\{e_i\}$ with $u = \sum a_i e_i$ and $v = \sum b_i e_i$ :

\langle u, v \rangle = \sum_{i=1}^n a_i \bar{b_i}

Remark 7.13a: Energy Interpretation

In signal processing, Parseval's identity says the total energy in time domain equals total energy in frequency domain. The Fourier coefficients $c_i$ represent energy at each frequency.

Example 7.23b: Energy in Fourier Series

For a function $f$ with Fourier coefficients $a_n, b_n$ :

\frac{1}{\pi}\int_{-\pi}^{\pi} |f(x)|^2\, dx = \frac{a_0^2}{2} + \sum_{n=1}^{\infty} (a_n^2 + b_n^2)

This is Parseval for the Fourier orthonormal basis.

Theorem 7.19b: Best Approximation Characterization

Let $\{e_1, \ldots, e_k\}$ be orthonormal. The vector $\hat{v} = \sum_{i=1}^k \langle v, e_i \rangle e_i$ is the best approximation to $v$ in $\text{span}\{e_i\}$ :

\|v - \hat{v}\| \leq \|v - w\| \quad \text{for all } w \in \text{span}\{e_i\}

Proof:

For any $w = \sum c_i e_i$ in the span:

\|v - w\|^2 = \|v\|^2 - 2\text{Re}\langle v, w \rangle + \|w\|^2

This is minimized when $c_i = \langle v, e_i \rangle$ , giving $w = \hat{v}$ .

∎

Remark 7.13b: Geometric Interpretation

The best approximation $\hat{v}$ is the orthogonal projection of $v$ onto the subspace spanned by $\{e_i\}$ . The residual $v - \hat{v}$ is orthogonal to the subspace.

Example 7.23c: Approximation Error

Approximate $v = (1, 2, 3, 4)^T$ using only $e_1 = (1,0,0,0)^T$ , $e_2 = (0,1,0,0)^T$ :

\hat{v} = \langle v, e_1 \rangle e_1 + \langle v, e_2 \rangle e_2 = (1, 2, 0, 0)^T

Error by Bessel: $\|v - \hat{v}\|^2 = \|v\|^2 - (1^2 + 2^2) = 30 - 5 = 25$

Theorem 7.19c: Parseval for Inner Products

For orthonormal basis $\{e_i\}$ with $u = \sum a_i e_i$ , $v = \sum b_i e_i$ :

\langle u, v \rangle = \sum_{i=1}^n a_i \bar{b_i}

This extends Parseval from norms to general inner products.

Remark 7.13c: Connection to Least Squares

Best approximation via orthogonal projection is the mathematical foundation of least squares:

Project data onto model space
Residual orthogonal to model space
Minimizes squared error

4. Inner Product in Orthonormal Coordinates

Theorem 7.20: Inner Product Formula

For orthonormal basis $\{e_i\}$ , if $u = \sum a_i e_i$ and $v = \sum b_i e_i$ :

\langle u, v \rangle = \sum_{i=1}^{n} a_i \bar{b_i}

Proof:

\langle u, v \rangle = \left\langle \sum_i a_i e_i, \sum_j b_j e_j \right\rangle = \sum_{i,j} a_i \bar{b_j} \langle e_i, e_j \rangle = \sum_i a_i \bar{b_i}

∎

Remark 7.14: Standard Inner Product

In orthonormal coordinates, the inner product becomes the standard dot product! This is why orthonormal bases are so convenient—they reduce all inner products to simple sums.

Example 7.24: Computing in Orthonormal Basis

In orthonormal basis, if $u = (1, 2, 3)^T$ and $v = (4, 5, 6)^T$ are coefficient vectors:

\langle u, v \rangle = 1(4) + 2(5) + 3(6) = 32

Example 7.24a: Complex Orthonormal Coordinates

In $\mathbb{C}^2$ with standard orthonormal basis, for $u = (1+i, 2)^T$ and $v = (1, i)^T$ :

\langle u, v \rangle = (1+i)\overline{1} + 2\overline{i} = (1+i) + 2(-i) = 1 - i

Theorem 7.20a: Norm Preservation

Orthonormal coordinates preserve norms: if $v = \sum c_i e_i$ , then:

\|v\|^2 = \|(c_1, \ldots, c_n)\|^2

The norm of $v$ equals the Euclidean norm of its coefficient vector.

Remark 7.14a: Isometry

The coordinate map $v \mapsto (\langle v, e_1 \rangle, \ldots, \langle v, e_n \rangle)$ is an isometry (distance-preserving map) between the inner product space and $\mathbb{C}^n$ .

Example 7.24b: Distance Calculation

Distance between $u = 2e_1 + 3e_2$ and $v = e_1 + 4e_2$ (orthonormal $\{e_i\}$ ):

\|u - v\| = \|(2-1, 3-4)\| = \|(1, -1)\| = \sqrt{2}

Theorem 7.20b: Angle Preservation

In orthonormal coordinates, the angle formula simplifies:

\cos\theta = \frac{\langle u, v \rangle}{\|u\| \|v\|} = \frac{\sum a_i \bar{b_i}}{\sqrt{\sum |a_i|^2} \sqrt{\sum |b_i|^2}}

Example 7.24c: Angle Calculation

Find angle between $u = (1, 1, 0)^T$ and $v = (0, 1, 1)^T$ in standard orthonormal basis:

\cos\theta = \frac{\langle u, v \rangle}{\|u\| \|v\|} = \frac{1}{\sqrt{2} \cdot \sqrt{2}} = \frac{1}{2}

So $\theta = \pi/3 = 60°$ .

Remark 7.14b: Cosine Similarity

In machine learning, cosine similarity is defined as:

\text{sim}(u, v) = \frac{\langle u, v \rangle}{\|u\| \|v\|}

This measures direction similarity regardless of magnitude, ranging from -1 (opposite) to +1 (same direction).

Theorem 7.20c: Orthonormal Coordinates are Unique Isometry

The map $\phi: V \to \mathbb{F}^n$ defined by $\phi(v) = (\langle v, e_1 \rangle, \ldots, \langle v, e_n \rangle)$ is a linear isometry:

$\phi$ is linear: $\phi(\alpha u + \beta v) = \alpha \phi(u) + \beta \phi(v)$
$\phi$ preserves inner product: $\langle \phi(u), \phi(v) \rangle = \langle u, v \rangle$
$\phi$ is bijective (isomorphism)

5. Extending Orthonormal Sets

Theorem 7.21: Orthonormal Extension

Any orthonormal set in a finite-dimensional inner product space can be extended to an orthonormal basis.

Proof:

Let $\{e_1, \ldots, e_k\}$ be orthonormal with $k < n = \dim(V)$ .

Since $\{e_i\}$ is linearly independent, extend to a basis $\{e_1, \ldots, e_k, v_{k+1}, \ldots, v_n\}$ .

Apply Gram-Schmidt to the new vectors, orthogonalizing against all previous ones.

∎

Example 7.25: Extending to Orthonormal Basis

Start with $e_1 = (1,0,0)^T$ in $\mathbb{R}^3$ . This is already unit length.

We can extend to $\{e_1, e_2, e_3\}$ where $e_2 = (0,1,0)^T$ and $e_3 = (0,0,1)^T$ .

Corollary 7.5: Dimension from Orthonormal Sets

Any orthonormal set with $n$ vectors in an $n$ -dimensional space is automatically a basis.

Example 7.25a: Extension in ℝ⁴

Extend $\{e_1 = \frac{1}{\sqrt{2}}(1,1,0,0)^T\}$ to orthonormal basis of $\mathbb{R}^4$ .

Step 1: Find vector orthogonal to $e_1$ , e.g., $e_2 = \frac{1}{\sqrt{2}}(1,-1,0,0)^T$

Step 2: Add $e_3 = (0,0,1,0)^T$ and $e_4 = (0,0,0,1)^T$

Verify all pairwise orthogonality and unit norms.

Theorem 7.21a: Orthonormal Basis for Orthogonal Complement

If $\{e_1, \ldots, e_k\}$ is orthonormal for subspace $U$ , extending to orthonormal basis $\{e_1, \ldots, e_n\}$ gives:

U^\perp = \text{span}\{e_{k+1}, \ldots, e_n\}

Proof:

Any $v \in U^\perp$ is orthogonal to $e_1, \ldots, e_k$ , so $v = \sum_{i=k+1}^n c_i e_i$ .

Conversely, any linear combination of $e_{k+1}, \ldots, e_n$ is orthogonal to $U$ .

∎

Example 7.25b: Finding Orthogonal Complement

In $\mathbb{R}^3$ , let $U = \text{span}\{(1,0,0)^T\}$ . Extend to basis:

\{(1,0,0)^T, (0,1,0)^T, (0,0,1)^T\}

Then $U^\perp = \text{span}\{(0,1,0)^T, (0,0,1)^T\}$ (the yz-plane).

Remark 7.15a: Dimension Formula

For a subspace $U$ of $V$ : $\dim(U) + \dim(U^\perp) = \dim(V)$

Theorem 7.21b: Direct Sum Decomposition

Every finite-dimensional inner product space decomposes as:

V = U \oplus U^\perp

Every $v$ can be uniquely written as $v = u + w$ with $u \in U$ , $w \in U^\perp$ .

Example 7.25c: Direct Sum in ℝ³

Let $U = \text{span}\{(1,1,0)^T\}$ . Then $U^\perp$ consists of vectors $(x, y, z)^T$ with $x + y = 0$ :

U^\perp = \text{span}\{(1,-1,0)^T, (0,0,1)^T\}

Decompose $v = (3, 1, 2)^T$ :

u = \text{proj}_U(v) = \frac{\langle v, (1,1,0)\rangle}{\|(1,1,0)\|^2}(1,1,0)^T = 2(1,1,0)^T

w = v - u = (1, -1, 2)^T \in U^\perp

Theorem 7.21c: Double Orthogonal Complement

For any subspace $U$ : $(U^\perp)^\perp = U$

Proof:

Clearly $U \subseteq (U^\perp)^\perp$ . By dimension counting:

\dim(U^\perp)^\perp = n - \dim(U^\perp) = n - (n - \dim U) = \dim U

So $U = (U^\perp)^\perp$ .

∎

Remark 7.15b: Orthogonal Complement of Sum

For subspaces $U, W$ :

(U + W)^\perp = U^\perp \cap W^\perp

(U \cap W)^\perp = U^\perp + W^\perp

These are analogs of De Morgan's laws for subspaces.

6. Orthonormal Bases in Function Spaces

Example 7.26: Fourier Basis

On $L^2[-\pi, \pi]$ , the set $\left\{ \frac{1}{\sqrt{2\pi}}, \frac{\cos(nx)}{\sqrt{\pi}}, \frac{\sin(nx)}{\sqrt{\pi}} \right\}_{n=1}^{\infty}$ is orthonormal.

This leads to Fourier series: any function can be expanded as:

f(x) = \frac{a_0}{2} + \sum_{n=1}^{\infty} (a_n \cos(nx) + b_n \sin(nx))

Example 7.27: Legendre Polynomials

On $L^2[-1,1]$ , the Legendre polynomials $P_n(x)$ (properly normalized) form an orthonormal basis:

P_0(x) = 1, \quad P_1(x) = x, \quad P_2(x) = \frac{1}{2}(3x^2 - 1), \ldots

Remark 7.15: Classical Orthogonal Polynomials

Different inner products (weight functions) give different orthogonal polynomial families:

Legendre: weight 1 on [-1,1]
Chebyshev: weight $1/\sqrt{1-x^2}$ on [-1,1]
Hermite: weight $e^{-x^2}$ on (-∞,∞)
Laguerre: weight $e^{-x}$ on [0,∞)

Example 7.27a: Verifying Fourier Orthogonality

Show $\sin(x)$ and $\cos(x)$ are orthogonal on $[-\pi, \pi]$ :

\langle \sin, \cos \rangle = \int_{-\pi}^{\pi} \sin(x)\cos(x)\, dx = \frac{1}{2}\int_{-\pi}^{\pi} \sin(2x)\, dx = 0

since $\sin(2x)$ is an odd function integrated over a symmetric interval.

Example 7.27b: Fourier Coefficients

For $f(x) = x$ on $[-\pi, \pi]$ , the Fourier sine coefficients are:

b_n = \frac{1}{\pi}\int_{-\pi}^{\pi} x \sin(nx)\, dx = \frac{2(-1)^{n+1}}{n}

So $x = 2\sum_{n=1}^{\infty} \frac{(-1)^{n+1}}{n} \sin(nx)$ for $x \in (-\pi, \pi)$ .

Theorem 7.22: Completeness of Fourier Basis

The Fourier system $\{1, \cos(nx), \sin(nx)\}_{n=1}^{\infty}$ is complete in $L^2[-\pi, \pi]$ : for any $f$ with $\|f\|_2 < \infty$ :

\lim_{N \to \infty} \left\|f - \sum_{n=-N}^{N} c_n e^{inx}\right\|_2 = 0

Remark 7.15b: Applications of Orthogonal Polynomials

Gaussian quadrature: Optimal numerical integration using zeros of orthogonal polynomials
Quantum mechanics: Hermite polynomials for harmonic oscillator, Laguerre for hydrogen atom
Approximation theory: Best polynomial approximation via orthogonal projection
Statistics: Orthogonal polynomial regression

Example 7.27c: Polynomial Approximation

Approximate $e^x$ on $[-1, 1]$ using Legendre polynomials:

e^x \approx c_0 P_0(x) + c_1 P_1(x) + c_2 P_2(x) + \cdots

where $c_n = \frac{2n+1}{2}\int_{-1}^{1} e^x P_n(x)\, dx$ .

Theorem 7.22a: Three-Term Recurrence

Classical orthogonal polynomials satisfy a recurrence relation:

P_{n+1}(x) = (a_n x + b_n) P_n(x) - c_n P_{n-1}(x)

This allows efficient computation without explicit integration.

Example 7.27d: Legendre Recurrence

Legendre polynomials satisfy:

(n+1)P_{n+1}(x) = (2n+1)xP_n(x) - nP_{n-1}(x)

With $P_0 = 1$ , $P_1 = x$ :

P_2 = \frac{3x^2 - 1}{2}, \quad P_3 = \frac{5x^3 - 3x}{2}

Remark 7.15c: Orthogonality in Infinite Dimensions

In infinite-dimensional Hilbert spaces:

Countably infinite orthonormal sets exist (e.g., Fourier basis)
Parseval extends: $\|v\|^2 = \sum_{i=1}^{\infty} |c_i|^2$ (may be infinite series)
Not all bases are orthonormal; Gram-Schmidt still works
Bessel's inequality always holds; Parseval characterizes completeness

Example 7.27e: Wavelet Bases

Wavelet bases provide orthonormal systems with localization:

\psi_{j,k}(x) = 2^{j/2}\psi(2^j x - k)

Used in signal processing, image compression (JPEG 2000), and denoising.

Theorem 7.22b: Riesz Representation

In a Hilbert space $H$ , every continuous linear functional $\phi: H \to \mathbb{F}$ has the form:

\phi(v) = \langle v, u \rangle

for a unique $u \in H$ . This connects functionals to vectors via the inner product.

7. Common Mistakes

Confusing orthogonal with orthonormal

Orthogonal = perpendicular. Orthonormal = perpendicular AND unit length. Always normalize!

Forgetting that zero vector is orthogonal to everything

⟨0, v⟩ = 0 for all v, but 0 cannot be in an orthogonal set (we require nonzero vectors for linear independence).

Using Parseval for non-complete sets

Parseval (equality) only holds for complete orthonormal bases. For partial sets, use Bessel (inequality).

Wrong Fourier coefficient formula

cₖ = ⟨v, eₖ⟩, NOT ⟨eₖ, v⟩ (matters for complex spaces due to conjugate symmetry).

Assuming orthonormal when only orthogonal

For orthogonal (not orthonormal) sets, the coefficient formula has a denominator: cⱼ = ⟨v, vⱼ⟩/||vⱼ||².

Forgetting to check all pairs

To verify orthogonality of n vectors, check all n(n-1)/2 pairs. Missing one pair invalidates the whole set.

Confusing orthogonal matrix with orthogonal set

An orthogonal matrix has orthonormal columns (not just orthogonal). The term "orthogonal matrix" is slightly misleading.

Applying Parseval to wrong space

Parseval requires the orthonormal set to be a basis for the ENTIRE space, not just a subspace.

Remark 7.16: Verification Checklist

Before using orthonormal basis properties:

✓ All vectors have unit length
✓ All pairs are orthogonal
✓ Set spans the desired space (if using Parseval)
✓ Using correct formula for real vs complex spaces

8. Key Results Summary

Orthonormal Properties

• ⟨eᵢ, eⱼ⟩ = δᵢⱼ
• Linearly independent
• Easy coefficient extraction
• Gram matrix = I

Key Identities

• Fourier: cₖ = ⟨v, eₖ⟩
• Parseval: ||v||² = Σ|cᵢ|²
• Bessel: Σ|⟨v,eᵢ⟩|² ≤ ||v||²
• Inner product: ⟨u,v⟩ = Σaᵢb̄ᵢ

Theorem Reference Table

Theorem	Statement	Condition
Linear Independence	Orthogonal nonzero → independent	Vectors nonzero
Fourier Coefficients	cₖ = ⟨v, eₖ⟩	Orthonormal basis
Parseval	\|\|v\|\|² = Σ\|cᵢ\|²	Complete ON basis
Bessel	Σ\|⟨v,eᵢ⟩\|² ≤ \|\|v\|\|²	Any ON set
Extension	ON set → ON basis	Finite dimension

Remark 7.17: Why Orthonormality Matters

Orthonormal bases are the "natural" coordinate systems for inner product spaces:

Computation: Coefficient extraction reduces to inner products (no matrix inversion)
Geometry: Distances and angles have simple formulas (standard dot product)
Analysis: Parseval connects norms in different representations
Numerics: Orthogonal transformations are stable (condition number = 1)

Theorem 7.23: Orthonormal Change of Basis

If $P$ is the matrix with orthonormal basis vectors as columns:

P^T P = I, \quad P^{-1} = P^T

Coordinate change is simply matrix-vector multiplication by $P^T$ .

Example 7.28: QR Decomposition Connection

Every matrix $A$ with independent columns can be written as $A = QR$ :

$Q$ : orthonormal columns (from Gram-Schmidt on columns of $A$ )
$R$ : upper triangular

This is the Gram-Schmidt process in matrix form!

9. Applications

Signal Processing

The Fourier orthonormal basis transforms signals between time and frequency domains:

Audio compression (MP3): Discard small Fourier coefficients
Noise filtering: Zero out high-frequency components
Feature extraction: Use coefficient magnitudes as features

\text{signal}(t) = \sum_n c_n e^{i\omega_n t} \quad \text{(synthesis)}

Quantum Mechanics

Quantum states live in Hilbert spaces with orthonormal bases:

Measurement: Probability = |⟨ψ, eₖ⟩|² (Born rule)
Observables: Self-adjoint operators with orthonormal eigenbases
Superposition: ψ = Σcₖeₖ with |Σ|cₖ|² = 1 (normalization)

Data Science and Machine Learning

PCA: Find orthonormal directions of maximum variance
SVD: Orthonormal bases for row and column spaces
Whitening: Transform data to have orthonormal covariance
Cosine similarity: ⟨u,v⟩/(||u||||v||) measures document similarity

Numerical Linear Algebra

Orthogonal transformations are preferred for numerical stability:

QR algorithm: Eigenvalue computation via orthogonal similarity
Householder reflections: Orthogonal matrices for stable factorizations
Givens rotations: Selective zeroing in sparse matrices
Condition number: κ(Q) = 1 for orthogonal Q (perfectly conditioned)

Example 7.29: Image Compression

The DCT (Discrete Cosine Transform) uses orthonormal cosine basis for JPEG:

F(u) = \sum_{x=0}^{N-1} f(x) \cos\left(\frac{\pi(2x+1)u}{2N}\right)

High-frequency coefficients (large $u$ ) are often small and can be discarded.

Computer Graphics

Orthonormal transformations are fundamental in 3D graphics:

Rotation matrices: SO(3) for 3D rotations preserving distances
Camera transforms: Orthonormal basis defines viewing direction
Normal vectors: Must be transformed by (A⁻¹)ᵀ, simplified if A orthogonal
Quaternions: Unit quaternions represent rotations via orthogonal matrices

Example 7.29a: 3D Rotation

Rotation by $\theta$ around z-axis:

R_z(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta & 0 \\ \sin\theta & \cos\theta & 0 \\ 0 & 0 & 1 \end{pmatrix}

Columns form orthonormal basis; $R_z^T R_z = I$ , $\det R_z = 1$ .

Communication Systems

OFDM (WiFi, 4G/5G): Orthogonal subcarriers prevent interference
CDMA: Orthogonal spreading codes for multiple access
Error correction: Orthogonal transforms in coding theory
Radar: Matched filters use orthogonal waveforms

Remark 7.20: Why Orthogonality is Fundamental

Orthogonality appears across mathematics and applications because:

Independence: Orthogonal components don't interfere
Stability: Orthogonal transformations preserve numerical accuracy
Efficiency: Coefficients computed by inner products (O(n) vs O(n³))
Optimality: Orthogonal projection gives best approximation

Orthogonality Practice

12

Questions

0

Correct

0%

Accuracy

1

Vectors

u, v

are orthogonal if:

Easy

Not attempted

2

An orthonormal set must satisfy:

Easy

Not attempted

3

Orthogonal nonzero vectors are:

Medium

Not attempted

4

For orthonormal basis

\{e_1,...,e_n\}

, coefficient

c_k

of

v = \sum c_i e_i

is:

Medium

Not attempted

5

Parseval's identity states:

Medium

Not attempted

6

The standard basis of

\mathbb{R}^n

is:

Easy

Not attempted

7

Bessel's inequality states that for orthonormal

\{e_i\}

:

Medium

Not attempted

8

To normalize vector

v

:

Easy

Not attempted

9

If

\{e_1, e_2\}

is orthonormal in

\mathbb{R}^3

:

Medium

Not attempted

10

The projection of

v

onto unit vector

u

is:

Medium

Not attempted

11

In an orthonormal basis, the inner product becomes:

Hard

Not attempted

12

Orthonormal bases are useful because:

Medium

Not attempted

Frequently Asked Questions

Why are orthonormal bases so useful?

Three main reasons: (1) Coefficients are trivial to compute as inner products, no matrix inversion needed. (2) The Gram matrix is the identity, simplifying all computations. (3) Parseval's identity gives ||v||² = Σ|cᵢ|², preserving norms.

Can every basis be made orthonormal?

Yes! The Gram-Schmidt process (next section) converts any basis to an orthonormal one spanning the same space. Every finite-dimensional inner product space has an orthonormal basis.

What's the difference between orthogonal and orthonormal?

Orthogonal: pairwise inner products are zero (⟨eᵢ,eⱼ⟩ = 0 for i≠j). Orthonormal: additionally, each vector has unit length (||eᵢ|| = 1). Orthonormal = orthogonal + normalized.

How do Fourier coefficients relate to orthonormal bases?

For orthonormal basis {eᵢ}, the Fourier coefficient cᵢ = ⟨v, eᵢ⟩ is the component of v along eᵢ. The expansion v = Σcᵢeᵢ is called the Fourier expansion. In function spaces, this gives Fourier series.

What is Bessel's inequality and when is it an equality?

Bessel: Σ|⟨v,eᵢ⟩|² ≤ ||v||² for any orthonormal set. It becomes Parseval's equality when {eᵢ} is a complete orthonormal basis spanning the space.

Why does orthogonality imply linear independence?

If Σcᵢeᵢ = 0, take inner product with eⱼ: cⱼ⟨eⱼ,eⱼ⟩ = 0 (other terms vanish by orthogonality). Since eⱼ ≠ 0, we have ||eⱼ||² > 0, so cⱼ = 0. This works for all j.

How many vectors can be mutually orthogonal in ℝⁿ?

At most n vectors can be mutually orthogonal in ℝⁿ (since orthogonal nonzero vectors are linearly independent, and dim(ℝⁿ) = n). An orthonormal basis achieves this maximum.

What's the geometric meaning of Fourier coefficients?

The Fourier coefficient ⟨v, eᵢ⟩ is the signed length of v's projection onto eᵢ. It tells you 'how much of v is in the eᵢ direction.' The vector v is reconstructed by summing these projections.

Do orthonormal bases exist in infinite dimensions?

Yes, but with subtleties. In Hilbert spaces, orthonormal bases exist and are countable. The Fourier basis {eⁱⁿˣ/√2π} is an orthonormal basis for L²[-π,π]. Parseval's identity extends to infinite sums.

How do I verify a set is orthonormal?

Check two things: (1) Each pair has zero inner product: ⟨eᵢ,eⱼ⟩ = 0 for i≠j. (2) Each vector has unit norm: ||eᵢ|| = 1. Equivalently, form the Gram matrix G with Gᵢⱼ = ⟨eᵢ,eⱼ⟩ and verify G = I.

10. What's Next

Building on Orthogonality

Gram-Schmidt Process (LA-7.3)

The algorithm to construct orthonormal bases from any basis. Converts any linearly independent set to an orthonormal one.

Orthogonal Projections (LA-7.4)

Project vectors onto subspaces using orthonormal bases. Foundation for least squares and best approximation.

Spectral Theorem (LA-7.5)

Self-adjoint operators have orthonormal eigenbases. The culmination of inner product space theory.

SVD (LA-8.1)

Singular value decomposition uses two orthonormal bases. The most important matrix factorization.

Remark 7.18: Study Tips

Practice verification: Given a set, quickly check orthonormality (all pairs, all norms)
Memorize the formulas: cₖ = ⟨v, eₖ⟩, Parseval, Bessel
Understand the geometry: Visualize orthogonality as perpendicularity in ℝ² and ℝ³
Connect to applications: See orthonormal bases in Fourier analysis, quantum mechanics, PCA

Example 7.30: Practice Problem 1

Let $v_1 = (1, 2, 2)^T$ and $v_2 = (2, 1, -2)^T$ . Verify orthogonality and normalize:

Solution:

\langle v_1, v_2 \rangle = 2 + 2 - 4 = 0 \quad \checkmark

\|v_1\| = \sqrt{1 + 4 + 4} = 3, \quad \|v_2\| = \sqrt{4 + 1 + 4} = 3

Orthonormal: $e_1 = \frac{1}{3}(1,2,2)^T, \quad e_2 = \frac{1}{3}(2,1,-2)^T$

Example 7.31: Practice Problem 2

Express $v = (5, 7, 4)^T$ in the orthonormal basis $\{e_1, e_2\}$ from above, plus $e_3 = \frac{1}{3}(2,-2,1)^T$ :

Solution:

c_1 = \langle v, e_1 \rangle = \frac{1}{3}(5 + 14 + 8) = 9

c_2 = \langle v, e_2 \rangle = \frac{1}{3}(10 + 7 - 8) = 3

c_3 = \langle v, e_3 \rangle = \frac{1}{3}(10 - 14 + 4) = 0

So $v = 9e_1 + 3e_2 + 0e_3$ . Verify Parseval: $9^2 + 3^2 + 0^2 = 90 = 25 + 49 + 16 = \|v\|^2$ ✓

Example 7.32: Practice Problem 3

Show that $\{1, x, x^2\}$ is NOT orthogonal on $[-1, 1]$ with standard inner product:

Solution:

\langle 1, x^2 \rangle = \int_{-1}^{1} x^2\, dx = \frac{2}{3} \neq 0

Not orthogonal! The Legendre polynomials are the orthogonal version.

Example 7.33: Practice Problem 4

Given orthonormal basis $\{e_1, e_2, e_3\}$ and $v = 2e_1 - e_2 + 3e_3$ , compute $\|v\|$ :

Solution:

\|v\|^2 = |2|^2 + |-1|^2 + |3|^2 = 4 + 1 + 9 = 14

So $\|v\| = \sqrt{14}$ .

Example 7.34: Practice Problem 5

Find $\langle u, v \rangle$ for $u = e_1 + 2e_2$ , $v = 3e_1 - e_2$ (orthonormal basis):

Solution:

\langle u, v \rangle = 1 \cdot 3 + 2 \cdot (-1) = 3 - 2 = 1

Remark 7.21: Key Skills Checklist

After this module, you should be able to:

✓ Verify if a set is orthogonal/orthonormal
✓ Compute Fourier coefficients in any orthonormal basis
✓ Apply Parseval's identity to find norms
✓ Use Bessel's inequality for approximation error
✓ Extend orthonormal sets to orthonormal bases
✓ Compute in orthonormal coordinates
✓ Understand orthogonal complements and direct sums

Pro Tips

• When in doubt, write vectors in orthonormal coordinates—formulas simplify dramatically
• Use Parseval to avoid computing norms directly from definitions
• Remember: orthonormal bases make the Gram matrix = I
• Bessel gives upper bounds even for incomplete orthonormal sets

11. Quick Reference

Definitions

Orthogonal set: ⟨vᵢ, vⱼ⟩ = 0 for i ≠ j
Orthonormal set: ⟨eᵢ, eⱼ⟩ = δᵢⱼ
Orthonormal basis: ON set that spans V
Fourier coefficient: cₖ = ⟨v, eₖ⟩
Orthogonal matrix: QᵀQ = I

Key Formulas

Expansion: v = Σ⟨v, eₖ⟩eₖ
Parseval: ||v||² = Σ|cₖ|²
Bessel: Σ|⟨v, eₖ⟩|² ≤ ||v||²
Inner product: ⟨u, v⟩ = Σaᵢb̄ᵢ
Pythagorean: ||Σvᵢ||² = Σ||vᵢ||²

Key Theorems

Independence: Orthogonal nonzero vectors are linearly independent
Extension: Any ON set extends to ON basis
Direct sum: V = U ⊕ U⊥
Dimension: dim(U) + dim(U⊥) = dim(V)

Common Examples

Standard basis: {eᵢ} in ℝⁿ, ℂⁿ
Fourier basis: {eⁱⁿˣ/√2π}
Legendre: Pₙ(x) on [-1,1]
Hermite: Hₙ(x) with e⁻ˣ² weight

Algorithm: Working with Orthonormal Bases

Given ON basis {e₁,...,eₙ}: To find coefficients of v, compute cₖ = ⟨v, eₖ⟩
To compute ⟨u, v⟩: Find coefficients aₖ, bₖ, then ⟨u, v⟩ = Σaₖb̄ₖ
To compute ||v||: Find coefficients cₖ, then ||v||² = Σ|cₖ|²
To verify ON: Check ⟨eᵢ, eⱼ⟩ = δᵢⱼ for all i, j
To extend ON set: Use Gram-Schmidt on additional vectors

Remark 7.19: Historical Note

Orthogonality concepts developed through several mathematical threads:

Fourier (1807): Trigonometric series for heat equation
Gram (1879), Schmidt (1907): Orthogonalization process
Hilbert (1906): Abstract infinite-dimensional spaces
von Neumann (1927): Quantum mechanics formulation using orthonormal bases

Notation Summary

$\langle \cdot, \cdot \rangle$ — inner product
$\|v\|$ — induced norm
$\delta_{ij}$ — Kronecker delta
$U^\perp$ — orthogonal complement

$c_k = \langle v, e_k \rangle$ — Fourier coefficient
$Q^T Q = I$ — orthogonal matrix
$U^H U = I$ — unitary matrix
$V = U \oplus U^\perp$ — direct sum

Quick Computation Guide

Find coefficient cₖ	Compute ⟨v, eₖ⟩
Find \|\|v\|\|	Use √(Σ\|cₖ\|²) (Parseval)
Find ⟨u, v⟩	Compute Σaₖb̄ₖ
Verify orthonormality	Check ⟨eᵢ, eⱼ⟩ = δᵢⱼ for all i, j
Find projection	Compute Σ⟨v, eₖ⟩eₖ

Orthogonality and Orthonormal Bases

1. Orthogonal and Orthonormal Sets

2. Orthonormal Bases

3. Parseval's Identity and Bessel's Inequality

4. Inner Product in Orthonormal Coordinates

5. Extending Orthonormal Sets

6. Orthonormal Bases in Function Spaces

7. Common Mistakes

Confusing orthogonal with orthonormal

Forgetting that zero vector is orthogonal to everything

Using Parseval for non-complete sets

Wrong Fourier coefficient formula

Assuming orthonormal when only orthogonal

Forgetting to check all pairs

Confusing orthogonal matrix with orthogonal set

Applying Parseval to wrong space

8. Key Results Summary

Orthonormal Properties

Key Identities

Theorem Reference Table

9. Applications

Frequently Asked Questions

Why are orthonormal bases so useful?

Can every basis be made orthonormal?

What's the difference between orthogonal and orthonormal?

How do Fourier coefficients relate to orthonormal bases?

What is Bessel's inequality and when is it an equality?

Why does orthogonality imply linear independence?

How many vectors can be mutually orthogonal in ℝⁿ?

What's the geometric meaning of Fourier coefficients?

Do orthonormal bases exist in infinite dimensions?

How do I verify a set is orthonormal?

10. What's Next

Building on Orthogonality

Gram-Schmidt Process (LA-7.3)

Orthogonal Projections (LA-7.4)

Spectral Theorem (LA-7.5)

SVD (LA-8.1)

Pro Tips

11. Quick Reference

Definitions

Key Formulas

Key Theorems

Common Examples

Algorithm: Working with Orthonormal Bases

Notation Summary

Quick Computation Guide