Cheatsheet: Matrices between dimensions
Reading the shape
Section titled “Reading the shape”An m x n matrix maps n-dimensional input to m-dimensional output.
| Count | Tells you |
|---|---|
n columns | Input dimension (one column per input basis vector) |
m rows | Output dimension (each column is an m-vector landing spot) |
Count the columns first: that is the input dimension.
Two directions
Section titled “Two directions”| Shape | Maps | Geometry |
|---|---|---|
| More rows than columns (e.g. 3x2) | small to big | Embedding: input space placed intact inside a bigger one |
| More columns than rows (e.g. 2x3) | big to small | Projection: input space squashed down (always crushes something) |
Where the spaces live
Section titled “Where the spaces live”- Column space = span of the columns = all reachable outputs. Lives in the output space.
- Null space = all inputs sent to zero. Lives in the input space.
- Rank = dimension of the column space, capped by the smaller of
mandn.
Conservation: rank + nullity = number of columns (the input dimension). A projection can be full rank and still have a null space, because the output is too small to keep every input distinct.
Worked examples
Section titled “Worked examples”| Matrix | Maps | Result | Rank | Null space |
|---|---|---|---|---|
[[1,0],[0,1],[1,1]] (3x2) | 2D to 3D | [3,4] -> [3,4,7] | 2 (a plane in 3D) | {0} |
[[1,0,0],[0,1,0]] (2x3) | 3D to 2D | [3,4,5] -> [3,4] | 2 (full) | z-axis (nullity 1) |
[[1,2],[2,4],[3,6]] (3x2) | 2D to 3D | columns dependent | 1 (a line in 3D) | line through [-2,1] |
All satisfy rank + nullity = number of columns.
Classify any rectangular matrix
Section titled “Classify any rectangular matrix”- Input dim = number of columns.
- Output dim = number of rows.
- Rank = dimension of the column space.
- Meaning: full-rank with more rows = embedding; more columns than rows = projection (has a null space); dependent columns = rank-deficient collapse.
Why it matters for AI
Section titled “Why it matters for AI”Neural network layers are non-square. A 256x768 matrix compresses a 768D embedding to 256D (projection); a 768x256 matrix expands 256D to 768D (embedding). Dimension reduction is rectangular matrices at work.
Pitfalls to dodge
Section titled “Pitfalls to dodge”- Reading the shape backward.
m x nmapsn-D in tom-D out. Columns = input dim. - Wrong side for the spaces. Column space in the output, null space in the input.
- Full rank means no null space. Only for square matrices. A projection keeps a null space.
- Determinant of a rectangle. Undefined. Use rank instead.
The one-line version
Section titled “The one-line version”A rectangular matrix carries one dimension count to another, embedding or projecting, and the same columns, rank, and null space describe what it does.