Gradient Descent

Gradient Descent is a mathematical optimization technique. Say we have a function: Z=wXZ=wX

that defines a line, which we want to estimate. The error between our guess and this line is j(w)=12m∑i=1m(zi−wxi)2j(w)=\frac{1}{2m}\sum_{i=1}^{m}(z_{i}-wx_i)^2

This is our cost function, our aim is to minimize it. If we plot this we get a parabola with a single point minimum that would be our solution.

We first take a random point w0w_0 Next we construct w1w_1from it with the following equation w1→w0−ηdjdww_1\to w_0 – \eta \frac{dj}{dw}

Now we construct w2w_2 in the same way. Eta η\eta is our learning rate. We can experiment with by increasing and decreasing this constant. In general we have this equation:

wi→wi−1−ηdjdww_i\to w_{i-1} – \eta \frac{dj}{dw}

Which we continue until we reach the optimal value of ww or the error is small enough for our needs.