ANN Series - 3 - Gradient Descent with Linear Activation

-

What is Gradient Descent?

Gradient Descent is a fundamental optimization algorithm used in machine learning and artificial intelligence to minimize the cost function, which is a measure of how far off a model's predictions are from the actual values. The goal of gradient descent is to find the model parameters (typically weights in neural networks) that minimize the cost function, thereby improving the model's predictions.

With numeric example, let us see how to find the model parameters using Gradient Descent where the activation function is the identity function f(x) = x.


Python Code for Calculating Gradient and Weight Updation:

ml-course/gd.ipynb at main · lovelynrose/ml-course (github.com)

How Gradient Descent Works: Summary

  1. Initialization: Start with initial values for the parameters to be optimized (weights in the case of neural networks).

  2. Compute Gradient: Calculate the gradient of the cost function with respect to each parameter. The gradient is a vector that represents the direction and rate of the fastest increase of the function. By computing the gradient, we find out how the cost function changes with changes in the parameters.

  3. Update Parameters: Adjust the parameters in the opposite direction of the gradient (the direction that reduces the cost function) by a small step. The size of the step is determined by the learning rate, a hyperparameter that controls how much we adjust the parameters with respect to the gradient. The formula for updating each parameter is:

    �=�−�×Gradient

    where
    �
    w represents the parameters,
    �

  4. Iterate: Repeat the process of calculating the gradient and updating the parameters until the cost function converges to a minimum value or until a specified number of iterations is reached.

Types of Gradient Descent:

  • Batch Gradient Descent: Uses the entire training dataset to calculate the gradient of the cost function for each iteration. While accurate, it can be very slow and computationally expensive for large datasets.

  • Stochastic Gradient Descent (SGD): Updates the parameters for each training example one by one. It is much faster and can be used for online learning, but the frequent updates result in a more fluctuating convergence path.

  • Mini-batch Gradient Descent: Strikes a balance between batch and stochastic gradient descent by updating parameters for a small subset of the training data at each iteration. This approach offers a compromise between the computational efficiency of SGD and the stability of batch gradient descent.

Key Features:

  • Convergence: Gradient descent aims to reach the global minimum of the cost function for convex problems or a local minimum for non-convex problems.

  • Learning Rate: Choosing the right learning rate is crucial. Too small a learning rate leads to slow convergence, while too large a learning rate can cause the algorithm to oscillate around the minimum or even diverge.

  • Application: Gradient descent is widely used in training a variety of machine learning models, including linear regression, logistic regression, and neural networks.

Gradient descent is powerful because it is simple yet effective at optimizing complex functions, making it a cornerstone of many machine learning algorithms.

Comments

Post a Comment

Popular posts from this blog

ANN Series - 10 - Backpropagation of Errors

-
Image
Watch the video to understand the forward pass in an ANN. Backpropagation, short for "backward propagation of errors," is a fundamental algorithm used for training artificial neural networks. It efficiently computes the gradient of the loss function with respect to the weights of the network, which is essential for adjusting the weights and minimizing the loss through gradient descent or other optimization techniques. The process involves two main phases: a forward pass and a backward pass.     Forward Pass   1. Input Layer:  The input features are fed into the network. 2. Hidden Layers:  Each neuron in these layers computes a weighted sum of its inputs (from the previous layer or the input layer) plus a bias term. This sum is then passed through an activation function to produce the neuron's output. This process repeats layer by layer until the output layer is reached. 3. Output Layer:  The final output of the network is computed, which is then used to calculate the loss
Read more

Regression 10 - Pseudo-Inverse Matrix Approach - Relationship between MLE and LSA - Derivation

-
Pseudo-Inverse Matrix Approach: The pseudo-inverse matrix approach is essentially the OLS approach. It provides a solution to the linear regression problem by minimizing the sum of squared residuals between the observed and predicted values. This approach is a direct algebraic method that does not make any probabilistic assumptions about the residuals; it simply finds the best fit in the least squares sense.   For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get from the pseudo-inverse matrix method, under the assumption of i.i.d. normal errors. In this blog we show how to prove that  Maximization of likelihood function under conditional Gaussian noise for a linear model is similar to the minimization of the sum of the squares error function.  The complete derivation along with formulae used are provided. Ensure that you go through the previous blogs to get a
Read more

Regression 9 - Maximum Likelihood Estimation(MLE) Approach for Regression - Derivation

-
  MLE Approach: When performing linear regression using MLE under the assumption that the residuals (errors) are independently and identically distributed (i.i.d.) with a normal distribution, we estimate the regression coefficients that maximize the likelihood of observing the given data. In this blog we see how to perform regression on a dataset that applies MLE for model fitting. The mathematical assumptions and derivations are given in detail.   The MLE approach gives you point estimates for the coefficients (mean of the likelihood distribution), and you can also compute the variance-covariance matrix of these estimates, which gives you the variances (and covariances) of the estimates. These variances are a measure of the uncertainty or the spread of the likelihood distribution of the parameter estimates. For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get
Read more