While this batching provides computation efficiency, it can still have a long processing time for large training datasets as it still needs to store all of the data into memory. This process referred to as a training epoch. Batch gradient descentīatch gradient descent sums the error for each point in a training set, updating the model only after all training examples have been evaluated. There are three types of gradient descent learning algorithms: batch gradient descent, stochastic gradient descent and mini-batch gradient descent. It’s worth noting that a loss function refers to the error of one training example, while a cost function calculates the average error across an entire training set. Additionally, while the terms, cost function and loss function, are considered synonymous, there is a slight difference between them.

At this point, the model will stop learning. It continuously iterates, moving along the direction of steepest descent (or the negative gradient) until the cost function is close to or at zero. This improves the machine learning model's efficacy by providing feedback to the model so that it can adjust the parameters to minimize the error and find the local or global minimum.

The cost (or loss) function measures the difference, or error, between actual y and predicted y at its current position.While it has the advantage of more precision, the number of iterations compromises overall efficiency as this takes more time and computations to reach the minimum. Conversely, a low learning rate has small step sizes. High learning rates result in larger steps but risks overshooting the minimum. This is typically a small value, and it is evaluated and updated based on the behavior of the cost function. Learning rate (also referred to as step size or the alpha) is the size of the steps that are taken to reach the minimum.These factors determine the partial derivative calculations of future iterations, allowing it to gradually arrive at the local or global minimum (i.e. In order to do this, it requires two data points-a direction and a learning rate. Similar to finding the line of best fit in linear regression, the goal of gradient descent is to minimize the cost function, or the error between predicted and actual y. The slope at the starting point will be steeper, but as new parameters are generated, the steepness should gradually reduce until it reaches the lowest point on the curve, known as the point of convergence. The slope will inform the updates to the parameters-i.e. From that starting point, we will find the derivative (or slope), and from there, we can use a tangent line to observe the steepness of the slope. The starting point is just an arbitrary point for us to evaluate the performance. The gradient descent algorithm behaves similarly, but it is based on a convex function.

You may also recall plotting a scatterplot in statistics and finding the line of best fit, which required calculating the error between the actual output and the predicted output (y-hat) using the mean squared error formula. You may recall the following formula for the slope of a line, which is y = mx + b, where m represents the slope and b is the intercept on the y-axis. Before we dive into gradient descent, it may help to review some concepts from linear regression.

Gradient formula