Two vectors are orthogonal exactly when their scalar product is 0.

The vectors must be pairwise orthogonal, i.e. they must be perpendicular to each other.

Two vectors are orthogonal, exactly when their scalar product is 0. But more is needed for them to be orthonormal.

A basis is orthonormal if the vectors are mutually orthogonal and also individually normalized, meaning that each of them has a length (norm) of 1.

You can rather easily guess a unit vector that is perpendicular to the given vector. Then find a third basis vector that is perpendicular to both a the cross product. Check that these basis vectors constitute an orthonormal basis with quick dot products.

If we want to find an orthonormal basis, we must first find any basis for the solution space, i.e. we must solve the equation.

With $x_2=t_1$ and $x_3=t_2$ we find that

What is the solution space? Meaning, what is a basis for the solution space?

The solution space can be stated as $\spanVec \lbrace (-1,1,0),(-1,0,1)\rbrace$ with basis vectors

When two vectors are to be orthonormalized, we can approach it by first rotating the vectors within the space they are spanning until they become orthogonal (mutually perpendicular) while still spanning the same space, and second by normalizing them, i.e. elongating or shortening them, so they all have a length of 1.

All is carried out via the Gram-Schmidt procedure.

See Definition 15.14 in eNote 15.

What are $\mA\cdot\mA\transp$, $\mB\cdot\mB\transp$, $\mC\cdot\mC\transp$ and $\mD\cdot\mD\transp?$

For an orthogonal matrix, $\mA^{-1}=\mA\transp.$

Compute $\mA\cdot\mA\transp.$ What must this equal for the matrix to be orthogonal?

$\mA$ is orthogonal if and only if $2a^2=1$ because $\mA\cdot\mA\transp$ must become the identity matrix for $\mA$ to be orthogonal.

Look at the discriminant of the characteristic polynomial. Can non-real solutions exist?

Find a matrix $\,\mV\,$ whose columns consist of eigenvectors of $\mA$.

A matrix consisting of eigenvectors as colunms is

which satisfies that $\mV^{-1}\cdot\mA\cdot\mV=\mathbf{\Lambda}$ when the diagonal matrix contains eigenvalues:

However, $\mV$ is not orthogonal yet.

$\mV$ can be orthogonalized via the Gram-Schmidt procedure on the columns.

Since $\mA$ is symmetric, the corresponding eigenspaces are orthogonal. So you only have to perform the Gram-Schmidt procedure on the two-dimensional eigenvector space. The one-dimensional eigenvector space just needs to be normalized.

$\mA$ and $\mB$ are symmetric. What is special about symmetrix matrices in this context?

For $\mA$, $\mv_1$ is orthogonal to the eigenvector space $E_2$. In the same way $\mv_2$ is orthogonal to the eigenvector space $E_3$ for $\mB$.

We can find the last eigenvector $\mv_3$, which they have in common, in the intersecting line between the two planes, $E_2$ for $\mA$ and $E_3$ for $\mB$, respectively, but can we do it more easily?

$\mv_3$ is perpendicular both to $\mv_1$ and $\mv_2.$

Therefore we can find $\mv_3$ as the cross product of $\mv_1$ and $\mv_2.$

Then the three vectors only need to be normalized.

The vector $\,\mv_1\,$, $\,\mv_2\,$ and $\,\mv_3\,$ constitute a basis for $\,\mathbf U\,,$ so they only need to be orthonormalized using Gram-Schmidt.

A complement (complemented subspace) to $\mathbf U$ contains the “remaining dimensions” of the vector space. What is the dimension of this complement?

Since $\mathbf U$ is a 3-dimensional subspace within $\reel^4$, then this complement is 1-dimensional. A single vector that is linearly independent from all vectors within $\mathbf U$ can be a basis for this complement.

When you have found such a basis vector for this complement to $\mathbf U$, then transform it to be orthogonal to $\mathbf U$ (to all vectors within $\mathbf U$). This can be done via the Gram-Schmidt procedure. Then you have the orthogonal complement.