Recall that if S={u1,...,up} is an orthogonal set of non-zero vectors, then S is linearly independent and forms a basis for the subspace spanned by S.
Theorem. Let {u1,...,up} be an orthogonal basis for a subspace W. For each y in W, the weights in the linear combination y=c1u1+...+cpup are given by cj=uj⋅ujy⋅uj for j=1,...,p.
This theorem lets us change basis without requiring any row reduction computations, as long as that basis is an orthogonal basis.
Consider a non-zero u in Rn. We want to write y=y^+z where y^=αu for some scalar α, and where z is orthogonal to u (that is, z⋅u=0).
zz⋅u000αy^=y−y^=y−αu=(y−αu)⋅u=(y−αu)⋅u=y⋅u−(αu)⋅u=y⋅u−α(u⋅u)=u⋅uy⋅u=u⋅uy⋅uu.
The vector y^ is called the orthogonal projection of y onto u, and the vector z is called the component of y orthogonal to u.
We can think of this as projecting y onto L, where L is the subspace spanned by the vector u, so sometimes y^ is denoted by projLy.
Essentially, y^ is the point in L that is closest to y, so this is where this notion of "projection" comes from. (Think of y^ as the "shadow" of y onto the "wall" L.)
In this way, you can also think of z as the shortest path between y and L.
Ex. Take y=(76), u=(42).
α=u⋅uy⋅u=2040=2y^=αu=2(42)=(84)z=y−y^=(−12)
Theorem. Let W be a subspace of Rn. Then each y in Rn can be written uniquely in the form y=y^+z, where y^ is in W and z is in W⊥.
In fact, if {u1,...,up} is any orthogonal basis of W, then projWy=y^=u1⋅u1y⋅u1u1+...+up⋅upy⋅upup and z=y−y^.
Ex. Let u1=⎝⎜⎛25−1⎠⎟⎞, u2=⎝⎜⎛−211⎠⎟⎞, y=⎝⎜⎛123⎠⎟⎞.
u1⋅u2=0, so u1 and u2 are orthogonal to each other, and as a result W={u1,u2} is an orthogonal basis for span(W).
projWy=y^=u1⋅u1y⋅u1u1+u2⋅u2y⋅u2u2=309⎝⎜⎛25−1⎠⎟⎞+63⎝⎜⎛−211⎠⎟⎞=⎝⎜⎛−52251⎠⎟⎞z=y−y^=⎝⎜⎛123⎠⎟⎞−⎝⎜⎛−52251⎠⎟⎞=⎝⎜⎛570514⎠⎟⎞
Definition. A collection of orthogonal vectors {u1,...,up} is said to be orthonormal if each ui in the collection is a unit vector (that is, ∥ui∥=1).
Theorem. If {u1,...,up} is an orthonormal basis for a subspace W of Rn, then projWy=(y⋅u1)u1+...+(y⋅un)un (since all the denominators ui⋅ui from the previous equation are now equal to 1).
Similarly, if u=[u1...up], then projWy=uuTy for all y in Rn (based on the definition of the dot product).