MATH 240

Wed. November 20th, 2019

Recall that if S={u1,...,up}S=\{u_1,...,u_p\} is an orthogonal set of non-zero vectors, then SS is linearly independent and forms a basis for the subspace spanned by SS.

Theorem. Let {u1,...,up}\{u_1,...,u_p\} be an orthogonal basis for a subspace WW. For each yy in WW, the weights in the linear combination y=c1u1+...+cpupy=c_1u_1+...+c_pu_p are given by cj=y⋅ujuj⋅ujc_j=\dfrac{y\cdot u_j}{u_j \cdot u_j} for j=1,...,pj=1,...,p.

This theorem lets us change basis without requiring any row reduction computations, as long as that basis is an orthogonal basis.

Orthogonal Projection

Consider a non-zero uu in Rn\R^n. We want to write y=y^+zy=\hat{y}+z where y^=αu\hat{y}=\alpha u for some scalar α\alpha, and where zz is orthogonal to uu (that is, z⋅u=0z\cdot u=0).

z=y−y^=y−αuz⋅u=(y−αu)⋅u0=(y−αu)⋅u0=y⋅u−(αu)⋅u0=y⋅u−α(u⋅u)α=y⋅uu⋅uy^=y⋅uu⋅uu.\begin{aligned} z&=y-\hat{y}\\ &=y-\alpha u\\ z\cdot u&=(y-\alpha u)\cdot u\\ 0&=(y-\alpha u)\cdot u\\ 0&=y\cdot u-(\alpha u)\cdot u\\ 0&=y\cdot u-\alpha(u\cdot u)\\ \alpha&=\dfrac{y\cdot u}{u\cdot u}\\\\ \hat{y}&=\dfrac{y\cdot u}{u\cdot u}u. \end{aligned}

The vector y^\hat{y} is called the orthogonal projection of yy onto uu, and the vector zz is called the component of yy orthogonal to uu.

We can think of this as projecting yy onto LL, where LL is the subspace spanned by the vector uu, so sometimes y^\hat{y} is denoted by projLy\text{proj}_L y.

Essentially, y^\hat{y} is the point in LL that is closest to yy, so this is where this notion of "projection" comes from. (Think of y^\hat{y} as the "shadow" of yy onto the "wall" LL.)
In this way, you can also think of zz as the shortest path between yy and LL.

Ex. Take y=(76)y=\begin{pmatrix} 7\\6 \end{pmatrix}, u=(42)u=\begin{pmatrix} 4\\2 \end{pmatrix}.

α=y⋅uu⋅u=4020=2y^=αu=2(42)=(84)z=y−y^=(−12)\begin{gathered} \alpha=\dfrac{y\cdot u}{u\cdot u}=\frac{40}{20}=2\\ \hat{y}=\alpha u=2\begin{pmatrix} 4\\2 \end{pmatrix}=\begin{pmatrix} 8\\4 \end{pmatrix}\\ z=y-\hat{y}=\begin{pmatrix} -1\\2 \end{pmatrix} \end{gathered}

The Orthogonal Decomposition Theorem

Theorem. Let WW be a subspace of Rn\R^n. Then each yy in Rn\R^n can be written uniquely in the form y=y^+z,y=\hat{y}+z, where y^\hat{y} is in WW and zz is in W⊥W^\perp.

In fact, if {u1,...,up}\{u_1,...,u_p\} is any orthogonal basis of WW, then projWy=y^=y⋅u1u1⋅u1u1+...+y⋅upup⋅upup\text{proj}_W y=\hat{y}=\dfrac{y\cdot u_1}{u_1\cdot u_1}u_1+...+\dfrac{y\cdot u_p}{u_p\cdot u_p}u_p and z=y−y^z=y-\hat{y}.

Ex. Let u1=(25−1)u_1=\begin{pmatrix} 2\\5\\-1 \end{pmatrix}, u2=(−211)u_2=\begin{pmatrix} -2\\1\\1 \end{pmatrix}, y=(123)y=\begin{pmatrix} 1\\2\\3 \end{pmatrix}.

u1⋅u2=0u_1\cdot u_2=0, so u1u_1 and u2u_2 are orthogonal to each other, and as a result W={u1,u2}W=\{u_1,u_2\} is an orthogonal basis for span(W)\text{span}(W).

projWy=y^=y⋅u1u1⋅u1u1+y⋅u2u2⋅u2u2=930(25−1)+36(−211)=(−25215)\begin{aligned} \text{proj}_W y=\hat{y}&=\frac{y\cdot u_1}{u_1\cdot u_1}u_1+\frac{y\cdot u_2}{u_2\cdot u_2}u_2\\ &=\frac{9}{30}\begin{pmatrix} 2\\5\\-1 \end{pmatrix}+\frac{3}{6}\begin{pmatrix} -2\\1\\1 \end{pmatrix}\\ &=\begin{pmatrix} -\frac{2}{5}\\2\\\frac{1}{5} \end{pmatrix} \end{aligned}z=y−y^=(123)−(−25215)=(750145)\begin{aligned} z&=y-\hat{y}\\ &=\begin{pmatrix} 1\\2\\3 \end{pmatrix}-\begin{pmatrix} -\frac{2}{5}\\2\\\frac{1}{5} \end{pmatrix}\\ &=\begin{pmatrix} \frac{7}{5}\\0\\\frac{14}{5} \end{pmatrix} \end{aligned}

Orthonormal Set

Definition. A collection of orthogonal vectors {u1,...,up}\{u_1,...,u_p\} is said to be orthonormal if each uiu_i in the collection is a unit vector (that is, ∥ui∥=1\lVert u_i \rVert=1).

Theorem. If {u1,...,up}\{u_1,...,u_p\} is an orthonormal basis for a subspace WW of Rn\R^n, then projWy=(y⋅u1)u1+...+(y⋅un)un\text{proj}_W y=(y\cdot u_1)u_1+...+(y\cdot u_n)u_n (since all the denominators ui⋅uiu_i\cdot u_i from the previous equation are now equal to 11).

Similarly, if u=[u1...up]u=[u_1 ... u_p], then projWy=uuTy\text{proj}_W y=uu^Ty for all yy in Rn\R^n (based on the definition of the dot product).