Naive Bayesian Classifiers - Multinomial, Bernoulli and Gaussian with Solved Examples and Laplace Smoothing

-

 Learn about the Naive Bayes Classifier in the following notes.

Numeric Example with Dataset (Transactional Data)

Sum1

Sum2

Consider the following dataset. Apply the Naïve Bayes classifier to the following frequency table and predict the type of fruit given it is {Yellow, Sweet, Long}.


Solution can be viewed in the following pdf.


Numeric Example with Text Data

Multinomial Naive Bayes vs Bernoulli Naive Bayes

Multinomial Naive Bayes and Bernoulli Naive Bayes are both variations of the Naive Bayes algorithm, and they are used for different types of data distributions:

1. Multinomial Naive Bayes:

   - The Multinomial Naive Bayes classifier is used for data that is multinomially distributed, which typically means it is used for discrete data.

   - It is particularly suitable for text classification problems where features (or words) can occur multiple times. For example, it can be used for document classification where the features are the frequencies of the words or n-grams within the documents.

   - The likelihood of the features is assumed to follow a multinomial distribution, which counts how often each outcome of a classification problem occurs.

The likelihood of observing a feature vector x = (x1, x2, ..., xn ) given a class c is calculated as:


2. Bernoulli Naive Bayes:

   - The Bernoulli Naive Bayes classifier is specifically designed for binary/boolean features, which means it is well-suited for data where features are independently and binarily distributed.

   - This model is useful when you’re working with binary feature vectors. For example, in text classification, instead of using word frequencies, you might use binary variables to indicate the presence or absence of a word.

   - The parameters that the model learns are the probabilities of a feature being present in a class, and it explicitly penalizes the non-occurrence of a feature that is indicative of the class, which is a significant distinction from Multinomial Naive Bayes.

The likelihood of observing a feature vector x = (x1, x2, ..., xn ) given a class c is calculated as:

In short, use Multinomial Naive Bayes when your feature data is counts (like word counts in text classification), and use Bernoulli Naive Bayes when your features are binary (0s and 1s to represent the presence or absence of a feature). The main difference lies in the distribution they assume for the input data and consequently how they calculate the likelihoods of the features.

Multinomial Naive Bayes

The following video solves a text classification problem using Multinomial Naive Bayes.

Laplace Smoothing

Laplace smoothing (also known as additive smoothing or Laplacian correction) is a technique used to handle the problem of zero probability in Naive Bayes classifiers. Without smoothing, if the training data has never seen a particular class-feature combination before, it will assign a probability of zero to this combination. This can be problematic because it will cause the entire probability of the class to be zero when using the Naive Bayes rule (since probabilities are multiplied together).

To avoid this, Laplace smoothing is applied by adding a small positive number to each count. The modified formula for calculating the probability of a feature \( f \) given a class \( c \) with Laplace smoothing is:

The effect of Laplace smoothing is to distribute some of the probability mass to unseen features to ensure that no feature has a probability of zero. This allows the classifier to make a prediction even when it encounters a previously unseen feature-class combination.

Bernoulli Naive Bayes

The following video solves a text classification problem using Bernoulli Naive Bayes.



Gaussian Naive Bayes

The following video solves a problem with Discrete and Continuous Features using Gaussian Naive Bayes.






Comments

Post a Comment

Popular posts from this blog

ANN Series - 10 - Backpropagation of Errors

-
Image
Watch the video to understand the forward pass in an ANN. Backpropagation, short for "backward propagation of errors," is a fundamental algorithm used for training artificial neural networks. It efficiently computes the gradient of the loss function with respect to the weights of the network, which is essential for adjusting the weights and minimizing the loss through gradient descent or other optimization techniques. The process involves two main phases: a forward pass and a backward pass.     Forward Pass   1. Input Layer:  The input features are fed into the network. 2. Hidden Layers:  Each neuron in these layers computes a weighted sum of its inputs (from the previous layer or the input layer) plus a bias term. This sum is then passed through an activation function to produce the neuron's output. This process repeats layer by layer until the output layer is reached. 3. Output Layer:  The final output of the network is computed, which is then used to calculate the loss
Read more

Regression 10 - Pseudo-Inverse Matrix Approach - Relationship between MLE and LSA - Derivation

-
Pseudo-Inverse Matrix Approach: The pseudo-inverse matrix approach is essentially the OLS approach. It provides a solution to the linear regression problem by minimizing the sum of squared residuals between the observed and predicted values. This approach is a direct algebraic method that does not make any probabilistic assumptions about the residuals; it simply finds the best fit in the least squares sense.   For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get from the pseudo-inverse matrix method, under the assumption of i.i.d. normal errors. In this blog we show how to prove that  Maximization of likelihood function under conditional Gaussian noise for a linear model is similar to the minimization of the sum of the squares error function.  The complete derivation along with formulae used are provided. Ensure that you go through the previous blogs to get a
Read more

Regression 9 - Maximum Likelihood Estimation(MLE) Approach for Regression - Derivation

-
  MLE Approach: When performing linear regression using MLE under the assumption that the residuals (errors) are independently and identically distributed (i.i.d.) with a normal distribution, we estimate the regression coefficients that maximize the likelihood of observing the given data. In this blog we see how to perform regression on a dataset that applies MLE for model fitting. The mathematical assumptions and derivations are given in detail.   The MLE approach gives you point estimates for the coefficients (mean of the likelihood distribution), and you can also compute the variance-covariance matrix of these estimates, which gives you the variances (and covariances) of the estimates. These variances are a measure of the uncertainty or the spread of the likelihood distribution of the parameter estimates. For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get
Read more