Summary: The cost landscape

Lesson 5 turned learning into a clean goal, make the cost small, but left us standing in a 13,000-dimensional space of knob settings with no idea which way to step. This lesson gives that space a shape. Picture the cost as a landscape, where each setting of the weights and biases is a point and its cost is the height above it, and “minimize the cost” becomes something almost physical: walk downhill. The compass that always points downhill has a name (the negative gradient), and finding it is what the next lesson turns into an algorithm. This is the scan-it-in-five-minutes version.

Core ideas

• Cost as terrain. Treat each knob setting as a point and its cost as a height: bad regions rise into hills, good regions sink into valleys, and the best setting is the bottom of the lowest valley. The landscape is a way of picturing C(w, b), not something the network stores.
• High dimensions are fine. A 2-knob landscape is a drawable 3D surface; a real network’s lives in 13,000+ dimensions, which nobody can picture. You do not need to: slopes, downhill, and valleys carry over to high dimensions as exact math. Build intuition in 2D and trust it.
• Every direction has a slope. Stand at a point and ask whether a tiny step makes the ground rise or fall; the answer depends on the direction. (This is the calculus derivative idea; Track 8 makes it precise.)
• The gradient points steepest uphill. The gradient ∇C is a direction (a vector), assembled from every knob’s individual slope, pointing the way cost rises fastest. The negative gradient -∇C points steepest downhill, and stepping that way lowers cost faster than any other step.
• Worked small. On C(w) = w² at w = 3, the slope is 2w = 6, so stepping toward 0 drops the cost from 9. On the bowl C(w1, w2) = w1² + w2² at (3, 4), the gradient is [6, 8] and the negative gradient [-6, -8] points back toward the bottom, where the cost falls from 25. Two knobs or thirteen thousand, downhill is always the negative gradient.
• Downhill reaches a local minimum, not always the global one. A point where every nearby step goes uphill is a local minimum (a valley bottom); the deepest valley anywhere is the global minimum. Downhill-only walking can settle in a shallow valley it cannot climb out of, so a trained network is usually a good solution, not a provably best one.

What changes for you

The landscape picture quietly demystifies several things about AI you may have noticed. Training is iterative and slow because reaching a valley takes many small downhill steps, not one calculation. Two runs of the same model can land on different results because they started at different random points and rolled into different valleys. “Converged” just means the downhill walk reached a valley floor where steps stopped lowering the cost much. None of it is mystical once you can see the terrain. The search now has a shape and a compass; lesson 7 finally takes the walk, starting somewhere, reading the negative gradient, stepping, and repeating, the algorithm this whole chapter is named for: gradient descent.