ANN Series 8 - Perceptron

-

In the previous blogs in the series we saw how to handle regression with a simple ANN architecture. In this blog, we perform classification with a single layered ANN for binary classification and it is termed perceptron. To update weights we make use of the Perceptron learning rule rather than Gradient Descent that was used in regression.

Perceptron

  • Binary classifier : Class = {1,-1}
  • Linear Classifier
  • Learns weight and threshold
  • Activation Function –
    • Step function –Binary or Sign
  • Let input vector
    • x = ( x1, x2, ... , xn)
  • Associate weight vector for the input
    • w = ( w1, w2, ... , wn)


 Perceptron Learning Algorithm

  • Step 1 : Initialize weights randomly
  • Step 2 : Take a training sample at random without replacement
  • Step 3 : Perform Affine transformation of input and weights
  • Step 4 : Pass output of affine transformation through activation function to find the class label
  • Step 5 : Update weights based on perceptron learning rule
  • Step 6 : Repeat from step 2 till all the inputs are correctly classified

 Refer previous blog for what is an affine transformation.




Geometric Intuition behind the Perceptron Learning Algorithm

Perceptron Criteria:
Case 1 : No misclassification
No update

Case 2 : Misclassified x_n
then Minimize -w . x_n * t_n
Since w.x=0, and since w and x are vectors, they must be orthogonal vectors. So the cosine angle between them must be 0. We want to move the decision boundary such that the decision boundary moves:
i) closer to the misclassified example when example is positive
ii) Farther from the misclassified example when example is negative

Intuitive Explanation

The weight vector can be thought of as pointing in the "ideal" direction where the sum of the weighted inputs gives the correct classification. Updating the weights based on misclassified examples gradually rotates and shifts this vector (and thus the decision boundary) towards a position where the classifications are as accurate as possible given the training data.

The bias update adjusts where the decision boundary intersects the feature space, allowing for fine-tuning of the boundary's position even when the inputs are all zeros.

In essence, the perceptron learning algorithm iteratively adjusts the orientation and position of the decision boundary by updating the weights and bias, aiming to minimize the classification errors on the training set.

Geometric Interpretation

  • The weight vector can be thought of as normal to the decision boundary. Changing rotates or tilts the boundary in the feature space.
  • The bias shifts the boundary closer or further away from the origin without rotating it.

Case 1: If x ε class:1 and w.x < 0  

     w_new = w+x

cos ϴ_new α w_new.x
                  α (w+x).x
                  α (w.x + x.x)
                  α (cos ϴ + x.x)
   cos ϴ_new > cos ϴ
          ϴ_new < ϴ

We see that with every updation, w_new moves towards x and the angle between the blue line and w is reducing. 

Case 2 : If x ε class:-1 and w.x ≥ 0  

     w_new = w-x

cos ϴ_new α w_new.x
                α (w-x).x
                  α (w.x - x.x)
                  α (cos ϴ - x.x)
   cos ϴ_new < cos ϴ
          ϴ_new > ϴ

We see that with every updation, w_new moves away from x and the angle between the red line and w is increasing. 

Limitations

  • Solves for Boolean output
  • Single layer perceptron solves only linearly separable problems cause they try to find a linear boundary
  • Updates weight and bias for every training sample

 




Comments

Post a Comment

Popular posts from this blog

ANN Series - 10 - Backpropagation of Errors

-
Image
Watch the video to understand the forward pass in an ANN. Backpropagation, short for "backward propagation of errors," is a fundamental algorithm used for training artificial neural networks. It efficiently computes the gradient of the loss function with respect to the weights of the network, which is essential for adjusting the weights and minimizing the loss through gradient descent or other optimization techniques. The process involves two main phases: a forward pass and a backward pass.     Forward Pass   1. Input Layer:  The input features are fed into the network. 2. Hidden Layers:  Each neuron in these layers computes a weighted sum of its inputs (from the previous layer or the input layer) plus a bias term. This sum is then passed through an activation function to produce the neuron's output. This process repeats layer by layer until the output layer is reached. 3. Output Layer:  The final output of the network is computed, which is then used to calculate the loss
Read more

Regression 10 - Pseudo-Inverse Matrix Approach - Relationship between MLE and LSA - Derivation

-
Pseudo-Inverse Matrix Approach: The pseudo-inverse matrix approach is essentially the OLS approach. It provides a solution to the linear regression problem by minimizing the sum of squared residuals between the observed and predicted values. This approach is a direct algebraic method that does not make any probabilistic assumptions about the residuals; it simply finds the best fit in the least squares sense.   For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get from the pseudo-inverse matrix method, under the assumption of i.i.d. normal errors. In this blog we show how to prove that  Maximization of likelihood function under conditional Gaussian noise for a linear model is similar to the minimization of the sum of the squares error function.  The complete derivation along with formulae used are provided. Ensure that you go through the previous blogs to get a
Read more

Regression 9 - Maximum Likelihood Estimation(MLE) Approach for Regression - Derivation

-
  MLE Approach: When performing linear regression using MLE under the assumption that the residuals (errors) are independently and identically distributed (i.i.d.) with a normal distribution, we estimate the regression coefficients that maximize the likelihood of observing the given data. In this blog we see how to perform regression on a dataset that applies MLE for model fitting. The mathematical assumptions and derivations are given in detail.   The MLE approach gives you point estimates for the coefficients (mean of the likelihood distribution), and you can also compute the variance-covariance matrix of these estimates, which gives you the variances (and covariances) of the estimates. These variances are a measure of the uncertainty or the spread of the likelihood distribution of the parameter estimates. For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get
Read more