Practice: Matrices between dimensions
Self-check
Section titled “Self-check”Six short questions. Answer each one in your head (or on paper) before opening the collapsible. Trying to retrieve the answer is where the learning sticks; rereading feels productive but does much less.
1. An m × n matrix maps input of what dimension to output of what dimension?
Show answer
It maps n-dimensional input to m-dimensional output. The number of columns (n) is the input dimension (one column per input basis vector), and the number of rows (m) is the output dimension (each column is an m-dimensional landing spot). Count the columns first.
2. What is the difference between an embedding and a projection?
Show answer
An embedding has more rows than columns (e.g. 3x2): it places a small input space intact inside a bigger output space, losing nothing. A projection has more columns than rows (e.g. 2x3): it squashes a big input space down into a smaller output, and always crushes something (it always has a null space).
3. Where do the column space and the null space each live, for a rectangular matrix?
Show answer
The column space (all reachable outputs, the span of the columns) lives in the output space. The null space (all inputs sent to zero) lives in the input space. For a 3x2 matrix, the column space is in 3D and the null space is in 2D.
4. Why does a projection always have a nontrivial null space, even at full rank?
Show answer
Because the output dimension is smaller than the input dimension, so there is not enough room in the output to keep every input distinct; something must be crushed. Full rank only forces a trivial null space when the matrix is square. A full-rank projection uses all of its (small) output and still collapses a direction.
5. What is the rank-nullity law for a rectangular matrix?
Show answer
rank + nullity = number of columns (the input dimension). The dimensions the transformation keeps plus the dimensions it crushes equal the input dimension. A 2x3 projection with rank 2 must have nullity 1, because 2 + 1 = 3.
6. Can you take the determinant of a rectangular matrix?
Show answer
No. Determinants are defined only for square matrices, because there is no single area or volume scaling factor when input and output dimensions differ. Rank is the right tool for describing what a rectangular matrix does.
Try it yourself, part 1: classify rectangular matrices
Section titled “Try it yourself, part 1: classify rectangular matrices”For each matrix, state the input dimension, the output dimension, the rank, the null space, and what kind of mapping it is (embedding, projection, or rank-deficient collapse). About 8 minutes, pen and paper.
- a)
[[1, 0], [0, 1], [2, 3]] - b)
[[1, 0, 2], [0, 1, 3]] - c)
[[1, 3], [2, 6], [1, 3]]
Check your work
- a) 2 columns, 3 rows: 2D in, 3D out. Columns
[1,0,2]and[0,1,3]are independent, so they span a plane in 3D: rank 2, null space{0}. An embedding (small space placed intact in a bigger one). - b) 3 columns, 2 rows: 3D in, 2D out. Columns
[1,0],[0,1],[2,3]; the first two already span all of 2D, so rank 2 (full). Null space:x + 2z = 0andy + 3z = 0givex = -2z,y = -3z, the line through[-2, -3, 1](check:M · [-2,-3,1] = [-2+2, -3+3] = [0,0]), nullity 1. A projection (2 + 1 = 3). - c) 2 columns, 3 rows: 2D in, 3D out. Columns
[1,2,1]and[3,6,3] = 3·[1,2,1]are dependent, so they span only a line: rank 1. Null space:x + 3y = 0, the line through[-3, 1](check:M · [-3,1] = [0,0,0]), nullity 1. Rank-deficient (a collapse onto a line in 3D;1 + 1 = 2).
Try it yourself, part 2: apply a rectangular matrix
Section titled “Try it yourself, part 2: apply a rectangular matrix”Apply each matrix to the given input using x · col1 + y · col2 (+ z · col3). About 6 minutes.
- a) The embedding
[[1, 0], [0, 1], [2, 3]]applied to[2, 1]. - b) The projection
[[1, 0, 2], [0, 1, 3]]applied to[2, 1, 1].
Check your work
- a) Columns
[1,0,2],[0,1,3].2 · [1,0,2] + 1 · [0,1,3] = [2,0,4] + [0,1,3] = [2, 1, 7]. A 2D input came out as a 3D point on the tilted image plane. - b) Columns
[1,0],[0,1],[2,3].2 · [1,0] + 1 · [0,1] + 1 · [2,3] = [2,0] + [0,1] + [2,3] = [4, 4]. A 3D input came out as a 2D point.
Flashcards
Section titled “Flashcards”Ten cards. Click any card to reveal the answer. Use the Print flashcards button to lay out the full set as one card per page, ready to print or save as a PDF for offline review.
Q. What does an m x n matrix map, dimension-wise?
n-dimensional input to m-dimensional output. The n columns are the input dimension (one per input basis vector); the m rows are the output dimension (each column is an m-vector landing spot).
Q. How do you read input vs output dimension from a matrix's shape?
Columns = input dimension, rows = output dimension. Count the columns first: that is how many basis vectors the input has. An m × n matrix goes from n-D to m-D.
Q. Embedding vs projection: which shape and which direction?
Embedding: more rows than columns (e.g. 3x2), small space placed intact inside a bigger one. Projection: more columns than rows (e.g. 2x3), big space squashed into a smaller one, always crushing something.
Q. Where do the column space and null space live for a rectangular matrix?
Column space (reachable outputs, span of the columns) lives in the output space. Null space (inputs sent to zero) lives in the input space. For a 3x2 matrix: column space in 3D, null space in 2D.
Q. Why does a projection always have a null space, even at full rank?
Its output dimension is smaller than its input dimension, so the output cannot keep every input distinct; a direction must be crushed. Full rank forces a trivial null space only for square matrices.
Q. What is the rank-nullity law for a rectangular matrix?
rank + nullity = number of columns (the input dimension). Dimensions kept plus dimensions crushed equal the input dimension. A rank-2 projection of 3D input must have nullity 1.
Q. What does rank-deficient mean for a rectangular matrix?
Its columns are linearly dependent, so they span fewer dimensions than they could. A 3x2 matrix whose two columns are parallel has rank 1: it collapses its 2D input onto a single line in 3D.
Q. Can you take the determinant of a rectangular matrix?
No. The determinant is defined only for square matrices, since there is no single area or volume scaling factor when input and output dimensions differ. Use rank for rectangular matrices.
Q. What caps the rank of an m x n matrix?
The smaller of m and n. A matrix cannot have more independent output directions than its output dimension, nor more than its number of columns. So a 2x3 matrix has rank at most 2.
Q. How do rectangular matrices show up in neural networks?
As the layers themselves. A 256x768 matrix compresses a 768-D embedding to 256-D (a projection); a 768x256 matrix expands 256-D to 768-D (an embedding). Dimension reduction is rectangular matrices at work.