Linear, Non-linear, Kernel Models

-

 Linear Parametric 

Ordinary Least Squares (OLS) and pseudo-inverse matrix methods are linear parametric models.

Ordinary Least Squares (OLS):

OLS is a linear regression method used for estimating the parameters of a linear model. In a simple linear regression, the model has the form

y=mx+b,

and OLS aims to find the values of m and b that minimize the sum of squared differences between the observed and predicted values of y. The linearity in this context refers to the linear combination of the model parameters.

Pseudo-inverse Matrix Method:

The pseudo-inverse matrix method is often used for solving linear systems of equations when the matrix is not square or invertible. In the context of linear regression, the pseudo-inverse is used to find the parameter vector in the equation

Y=Xβ+ε,

where Y is the response variable, X is the design matrix of predictors, β is the parameter vector, and ε is the error term. The solution involves finding the pseudo-inverse of the matrix X and multiplying it by the response vector Y.

While both OLS and the pseudo-inverse matrix method are linear methods, it's important to note that they assume a linear relationship between the predictors and the response variable. If the underlying relationship is non-linear, these methods may not perform well, and alternative approaches like kernel methods or non-linear regression models might be more appropriate.

Kernel Methods:

Support Vector Machines (SVM) with Non-Linear Kernels:

SVM is a powerful classification and regression technique. When using a non-linear kernel, such as the Gaussian (Radial Basis Function) kernel, SVM can effectively model complex, non-linear relationships in the data.

Kernelized Ridge Regression:

Ridge regression is a linear regression method with a regularization term. By using kernel trick, you can extend it to handle non-linear relationships.

Non-Linear Regression Models:

Polynomial Regression:

Instead of fitting a straight line, polynomial regression fits a polynomial function to the data. The model has the form y=β0 +β1 x+β2x2 +…+βn xn , allowing for non-linear relationships.

Exponential Regression:

In exponential regression, the model has the form , capturing relationships that grow or decay exponentially.

Neural Networks:

Deep learning models, particularly neural networks, are powerful tools for capturing complex non-linear relationships in data. They consist of interconnected layers of nodes and can be designed to learn intricate patterns.

Parametric Models

They store the parameter only and not the entire training dataset.

AI Generated

Comments

Post a Comment

Popular posts from this blog

ANN Series - 10 - Backpropagation of Errors

-
Image
Watch the video to understand the forward pass in an ANN. Backpropagation, short for "backward propagation of errors," is a fundamental algorithm used for training artificial neural networks. It efficiently computes the gradient of the loss function with respect to the weights of the network, which is essential for adjusting the weights and minimizing the loss through gradient descent or other optimization techniques. The process involves two main phases: a forward pass and a backward pass.     Forward Pass   1. Input Layer:  The input features are fed into the network. 2. Hidden Layers:  Each neuron in these layers computes a weighted sum of its inputs (from the previous layer or the input layer) plus a bias term. This sum is then passed through an activation function to produce the neuron's output. This process repeats layer by layer until the output layer is reached. 3. Output Layer:  The final output of the network is computed, which is then used to calculate the loss
Read more

Regression 10 - Pseudo-Inverse Matrix Approach - Relationship between MLE and LSA - Derivation

-
Pseudo-Inverse Matrix Approach: The pseudo-inverse matrix approach is essentially the OLS approach. It provides a solution to the linear regression problem by minimizing the sum of squared residuals between the observed and predicted values. This approach is a direct algebraic method that does not make any probabilistic assumptions about the residuals; it simply finds the best fit in the least squares sense.   For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get from the pseudo-inverse matrix method, under the assumption of i.i.d. normal errors. In this blog we show how to prove that  Maximization of likelihood function under conditional Gaussian noise for a linear model is similar to the minimization of the sum of the squares error function.  The complete derivation along with formulae used are provided. Ensure that you go through the previous blogs to get a
Read more

Regression 9 - Maximum Likelihood Estimation(MLE) Approach for Regression - Derivation

-
  MLE Approach: When performing linear regression using MLE under the assumption that the residuals (errors) are independently and identically distributed (i.i.d.) with a normal distribution, we estimate the regression coefficients that maximize the likelihood of observing the given data. In this blog we see how to perform regression on a dataset that applies MLE for model fitting. The mathematical assumptions and derivations are given in detail.   The MLE approach gives you point estimates for the coefficients (mean of the likelihood distribution), and you can also compute the variance-covariance matrix of these estimates, which gives you the variances (and covariances) of the estimates. These variances are a measure of the uncertainty or the spread of the likelihood distribution of the parameter estimates. For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get
Read more