Data → Machine Learning & AI
Gradient Descent
An iterative optimization method that updates parameters in the direction that reduces a loss function.
Motivation
Gradient descent solves the problem of improving parameters by repeatedly moving in the direction that reduces loss.
Where it fits
It belongs to optimization and model training.
Mental model
Imagine standing on a landscape and walking downhill. The gradient tells you which direction is uphill, so you step the other way.
Important details
Important details include learning rate, local minima, stochastic batches, and convergence behavior.
Common mistakes
- Using a learning rate that is too high or too low.
- Assuming optimization success means the model generalizes.