ANN Series 5 - Sigmoid Activation Function - First order derivative of sigmoid and tanh

-

Linear Activation Functions

  • For classification problems, linear functions like the identity function can have Separating Plane  to be linear hyperplane. With a single layer of ANN can only it can produce a line.
  • A model composed of only linear functions, no matter how many layers it has, cannot learn non-linear boundaries because the composition of linear functions is itself a linear function. However, the introduction of non-linear activation functions between layers enables neural networks to learn non-linear separators.
  • They are suitable for regression tasks, especially in the output layer for problems expecting a continuous output range.

Sigmoid Functions

Sigmoid Functions are mathematical function with S shaped curve(sigmoid curve)

Types of Sigmoid functions

1. Logistic function

Output between 0 and 1
f(x) = 1/(1+𝑒^(−𝑥) )

2. Hyperbolic tangent

Output between -1 and 1
f(x) = (𝑒^𝑥−𝑒^(-𝑥))/(𝑒^𝑥+𝑒^(-𝑥))

Effect of w and b on logistic function

When we use the logistic function it gives an S-Shaped curve as shown below.



The graph above demonstrates how different weights (w) and biases (b) affect the structure of the sigmoid function. Each curve represents a sigmoid function with a specific combination of weight and bias:

Weight(w) :

Determines the slope of the sigmoid curve. A larger weight makes the transition from 0 to 1 steeper, indicating that the function becomes more sensitive to changes in the input (x). This is evident as we compare the curves for (w=0.5), (w=1), and (w=2), where the steepness increases with the weight.

Bias(b) : 

Shifts the curve left or right along the (x)-axis. A positive bias shifts the curve to the left, meaning the sigmoid function reaches the transition region (where it goes from near 0 to near 1) at lower input values. Conversely, a negative bias shifts the curve to the right. This effect is shown by the different positions of the curves for biases (b=-2), (b=0), and (b=2).

By adjusting the weight and bias, one can control the sensitivity and the threshold at which the sigmoid function activates, making these parameters crucial for training neural networks to fit various data patterns.

Python Code:

The Python code used to produce the graph can be found here:
ml-course/sigmoid.ipynb at main · lovelynrose/ml-course (github.com)

First Order Derivative of the Logistic Function

First Order Derivative of the tanh Function


Effect on Gradient Descent

The gradient descent algorithm requires the first order derivative of the activation function to make the weight updation. We see that the first order derivative of the sigmoid functions can be written in terms of the sigmoid functions. This means that once you have computed the sigmoid function's value for any given input x, you can easily compute its derivative without going through the differentiation process again. 

Example

For x=0.5. the values of the logistic (sigmoid) function and the hyperbolic tangent (tanh⁡) functions are:

Logistic (Sigmoid) value: 0.6224593312018546

Hyperbolic Tangent (tanh⁡) value: 0.46211715726000974

For x=0.5, the first-order derivatives of the logistic (sigmoid) function and the hyperbolic tangent (tanh) function are:

Derivative of the Logistic (Sigmoid) function: 0.2350037122015945

We can derive the same answer by substituting in:

f'(x) = f(x)(1-f(x)

= 0.6224593312018546 (1- 0.6224593312018546 )

= 0.2350037122015945

Derivative of the Hyperbolic Tangent (tanh⁡) function: 0.7864477329659274

We get the same answer by substituting in the formula:

tanh'(x) = 1-tanh(x)^2

= 1 - 0.46211715726000974^ 2

= 0.7864477329659274


Comments

Post a Comment

Popular posts from this blog

ANN Series - 10 - Backpropagation of Errors

-
Image
Watch the video to understand the forward pass in an ANN. Backpropagation, short for "backward propagation of errors," is a fundamental algorithm used for training artificial neural networks. It efficiently computes the gradient of the loss function with respect to the weights of the network, which is essential for adjusting the weights and minimizing the loss through gradient descent or other optimization techniques. The process involves two main phases: a forward pass and a backward pass.     Forward Pass   1. Input Layer:  The input features are fed into the network. 2. Hidden Layers:  Each neuron in these layers computes a weighted sum of its inputs (from the previous layer or the input layer) plus a bias term. This sum is then passed through an activation function to produce the neuron's output. This process repeats layer by layer until the output layer is reached. 3. Output Layer:  The final output of the network is computed, which is then used to calculate the loss
Read more

Regression 10 - Pseudo-Inverse Matrix Approach - Relationship between MLE and LSA - Derivation

-
Pseudo-Inverse Matrix Approach: The pseudo-inverse matrix approach is essentially the OLS approach. It provides a solution to the linear regression problem by minimizing the sum of squared residuals between the observed and predicted values. This approach is a direct algebraic method that does not make any probabilistic assumptions about the residuals; it simply finds the best fit in the least squares sense.   For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get from the pseudo-inverse matrix method, under the assumption of i.i.d. normal errors. In this blog we show how to prove that  Maximization of likelihood function under conditional Gaussian noise for a linear model is similar to the minimization of the sum of the squares error function.  The complete derivation along with formulae used are provided. Ensure that you go through the previous blogs to get a
Read more

Regression 9 - Maximum Likelihood Estimation(MLE) Approach for Regression - Derivation

-
  MLE Approach: When performing linear regression using MLE under the assumption that the residuals (errors) are independently and identically distributed (i.i.d.) with a normal distribution, we estimate the regression coefficients that maximize the likelihood of observing the given data. In this blog we see how to perform regression on a dataset that applies MLE for model fitting. The mathematical assumptions and derivations are given in detail.   The MLE approach gives you point estimates for the coefficients (mean of the likelihood distribution), and you can also compute the variance-covariance matrix of these estimates, which gives you the variances (and covariances) of the estimates. These variances are a measure of the uncertainty or the spread of the likelihood distribution of the parameter estimates. For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get
Read more