Parametric and Non-parametric Models

-

Parametric models

Parametric models in machine learning refer to a class of models that make assumptions about the function form or the underlying data distribution. These models are characterized by a finite number of parameters, which means that irrespective of the size of the data, the complexity of the model is fixed. The goal of learning in parametric models is to estimate these parameters from the data. Once the parameters are learned, the model can make predictions for new, unseen data. 

Here are some key points about parametric models:

1. Fixed Number of Parameters: 

The number of parameters is predetermined before the training process begins. This number does not grow as the size of the training data increases.

2. Assumptions About Data Distribution: 

Parametric models often make strong assumptions about the form of the data distribution. For example, a linear regression model assumes that the relationship between the dependent and independent variables is linear.

 3. Simplicity and Efficiency: 

Due to their fixed complexity, parametric models can be more straightforward and faster to train compared to non-parametric models, especially when dealing with large datasets.

 4. Risk of Underfitting: 

Because of their simplicity and the assumptions they make about the data distribution, there's a risk that parametric models may not capture the true underlying patterns in the data, leading to underfitting.

 5. Examples of Parametric Models: 

Some common examples of parametric models include linear regression, logistic regression, and naive Bayes classifiers. Each of these models relies on a specific functional form and estimates parameters within that framework.

Parametric models are widely used in many areas of machine learning due to their simplicity, interpretability, and efficiency in training and prediction, especially when the underlying assumptions about the data distribution are reasonable or when computational resources are limited.

Non-parametric models

Non-parametric models in machine learning are a category of models that do not make explicit assumptions about the form or the distribution of the underlying data. Unlike parametric models, which have a fixed number of parameters determined prior to training, non-parametric models have a flexibility that allows them to adapt to the complexity of the data. This means that the number of parameters in non-parametric models can grow as more data is observed, allowing these models to potentially capture more complex patterns in the data. 

Here are some key aspects of non-parametric models:

1. Flexibility: 

Non-parametric models are highly flexible as they do not assume a particular functional form. This allows them to model complex and non-linear relationships in the data more effectively than many parametric models.

2. Data-Driven: 

These models allow the data to speak for itself since their structure can grow with the data. As a result, non-parametric models can be more accurate when the true relationship between variables is unknown or highly complex.

3. Risk of Overfitting: 

While the flexibility of non-parametric models is a strength, it can also be a weakness. Without proper regularization or stopping criteria, these models are prone to overfitting, especially with small datasets or datasets with a lot of noise.

4. Computationally Intensive: 

As the amount of data increases, non-parametric models can become computationally expensive both in terms of memory and processing power because their complexity grows with the data.

5. Examples of Non-Parametric Models: 

Some common examples of non-parametric models include k-nearest neighbors (k-NN), decision trees (and by extension, random forests), and kernel density estimators. Another notable example is the Gaussian process, used in regression and classification problems.

 Non-parametric models are particularly useful in situations where little is known about the underlying data distribution, or when the relationship between the input features and the output is expected to be complex or non-linear. They are widely used in classification, regression, and clustering tasks across various fields such as finance, healthcare, and image processing.

 The choice between parametric and non-parametric models often depends on several factors, including the size of the dataset, the complexity of the problem, computational resources, and the specific goals of the analysis. Balancing the strengths and weaknesses of each model type is crucial in selecting the most appropriate approach for a given machine learning task.

 Parametric Models

 1. Linear Regression

2. Logistic Regression

3. Polynomial Regression

4. Naive Bayes

5. Perceptron

6. Linear Discriminant Analysis (LDA)

7. Generalized Linear Models (GLM)

8. Support Vector Machines (SVM) with linear kernel

  Non-Parametric Models

 1. k-Nearest Neighbors (k-NN)

2. Decision Trees

3. Random Forests

4. Gradient Boosting Machines (e.g., XGBoost, LightGBM)

5. Kernel SVM

6. Gaussian Processes

7. Neural Networks (considered non-parametric due to their ability to model complex functions with a large number of parameters)

8. Density-Based Spatial Clustering of Applications with Noise (DBSCAN)

AI Generated


Comments

Post a Comment

Popular posts from this blog

ANN Series - 10 - Backpropagation of Errors

-
Image
Watch the video to understand the forward pass in an ANN. Backpropagation, short for "backward propagation of errors," is a fundamental algorithm used for training artificial neural networks. It efficiently computes the gradient of the loss function with respect to the weights of the network, which is essential for adjusting the weights and minimizing the loss through gradient descent or other optimization techniques. The process involves two main phases: a forward pass and a backward pass.     Forward Pass   1. Input Layer:  The input features are fed into the network. 2. Hidden Layers:  Each neuron in these layers computes a weighted sum of its inputs (from the previous layer or the input layer) plus a bias term. This sum is then passed through an activation function to produce the neuron's output. This process repeats layer by layer until the output layer is reached. 3. Output Layer:  The final output of the network is computed, which is then used to calculate the loss
Read more

Regression 10 - Pseudo-Inverse Matrix Approach - Relationship between MLE and LSA - Derivation

-
Pseudo-Inverse Matrix Approach: The pseudo-inverse matrix approach is essentially the OLS approach. It provides a solution to the linear regression problem by minimizing the sum of squared residuals between the observed and predicted values. This approach is a direct algebraic method that does not make any probabilistic assumptions about the residuals; it simply finds the best fit in the least squares sense.   For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get from the pseudo-inverse matrix method, under the assumption of i.i.d. normal errors. In this blog we show how to prove that  Maximization of likelihood function under conditional Gaussian noise for a linear model is similar to the minimization of the sum of the squares error function.  The complete derivation along with formulae used are provided. Ensure that you go through the previous blogs to get a
Read more

Regression 9 - Maximum Likelihood Estimation(MLE) Approach for Regression - Derivation

-
  MLE Approach: When performing linear regression using MLE under the assumption that the residuals (errors) are independently and identically distributed (i.i.d.) with a normal distribution, we estimate the regression coefficients that maximize the likelihood of observing the given data. In this blog we see how to perform regression on a dataset that applies MLE for model fitting. The mathematical assumptions and derivations are given in detail.   The MLE approach gives you point estimates for the coefficients (mean of the likelihood distribution), and you can also compute the variance-covariance matrix of these estimates, which gives you the variances (and covariances) of the estimates. These variances are a measure of the uncertainty or the spread of the likelihood distribution of the parameter estimates. For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get
Read more