Quiz 3 (Unit III) Study Guide

• Course Home
• Syllabus
• Calendar
• Readings
• MATLAB Exercises
• Problem Sets
• Study Materials

« Back: Study Materials

Quiz 3 is on the material of Unit III (though of course your MATLAB^® knowledge should be cumulative). In preparation for Quiz 3 you should find the study guide/summary below useful.

Note we anticipate that there will be several questions which involve linear algebra vector and matrix operations, several questions which involve least squares, and several questions which involve regression and statistical inference. There will be four questions or at most five questions in total, each with multiple parts. MATLAB will be sprinkled throughout, as in past quizzes.

MATLAB

Matrix and vector operations: inner product, norm (and corresponding MATLAB built-in "norm"), transpose (the ' operator), "inv" (MATLAB built-in to find inverse matrix), "eye" (MATLAB built-in for identity matrix), matrix vector product, matrix matrix product. Understand the difference between dotted operators and un-dotted operators. Formation of regression matrix X in terms of columns. The MATLAB backslash: application to solution of a system of linear equations; even more importantly, application to solution of least squares problems.

Linear Algebra Operations

Vectors and matrices: definition; rows and columns of a matrix; a matrix as n m-by-1 (column) vectors or m 1-by-n (row) vectors; the transpose operation.

Vector operations: scaling; addition; the inner product ("with two hands").

Matrix operations: scaling; addition; and multiplication. Matrix vector products from both two-handed (inner product) viewpoint and one-handed (weighted sum of columns) viewpoint. Requirements for matrix multiplication: agreement on inner dimension. Rules for matrix multiplication: A*B*C = A*(B*C) = (A*B)*C, A*(B+C) = A*B + A*C, (A*B)' = B'*A', but in general A*B does NOT equal B*A (even if both products can be formed).

Definition of the identity matrix I. Definition of the inverse matrix for a non-singular matrix A (inv(A)*A = A*inv(A) = Identity Matrix). Formula for the inverse of a 2-by-2 non-singular matrix.

Least Squares and Regression

The general model of Section 17.2.1: x, p, y, beta_j, h_j, n, m; m-by-1 vector of measurements Y, and m-by-n regression matrix X; fundamental relationship (X*beta)_i = Y_model(x_i;beta), for i = 1,...,m.

Notion of "bias-free" and beta^{true}. (We will only consider bias-free models in the quiz.) Understand when X*beta = Y has a solution (i.e., when Y is noise free) and when X*beta = Y in general has no solution (i.e., when m > n and Y is noisy). Recall the simple example of putting a straight line through m points.

The residual vector r(beta) = Y - X*beta as "misfit" or "misprediction"; the least squares formulation — betahat minimizes norm( r(beta) ) squared over all possible beta; the normal equations for betahat — (X'*X)*betahat = X'*Y, which can be solved in MATLAB as betahat = X \ Y.

Assumptions on noise: normal and zero mean (N1); homoscedastic (N2) with std dev sigma (a scalar); uncorrelated (N3).

Regression estimates: estimate betahat for beta^{true}; estimate sigmahat (in terms of norm(r(betahat)) = norm(Y - X*betahat) for sigma; "reason" for m - n rather than just m in the sigmahat expression; interpretation of betahat as "best explanation of data" and of sigmahat as "what is 'left over'"; joint confidence intervals for the beta^{true} in terms of the n-by-n matrix SIGMAhat and s_{gamma = 0.95, k, q} from Table 17.1(b); frequentist interpretation of joint confidence intervals; "hypothesis test" interpretation of joint confidence intervals (i.e., conclusions about coefficients beta^{true}). Understand why confidence intervals shrink as m increases (e.g., the case of n = 1 discussed in class).

Overfitting: why it is evil, why m should be large compared to n, and how overfitting is reflected in confidence intervals. The Goldilocks Principle.