Regression 7. Understanding Non-linear Regression with Polynomial Curve fitting

-

 Polynomial Curve Fitting

Polynomial curve fitting is a statistical technique used to model a relationship between a dependent variable and one or more independent variables using a polynomial equation. This approach is particularly useful when the relationship between the variables is non-linear and cannot be adequately described by a simple linear model.

Example

Create a dataset with one independent variable x and a dependent variable denoted by sin(2πx).

x

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

sin(2πx)

0

0.5878

0.9511

0.9511

0.5878

0

-0.5878

-0.9511

-0.9511

-0.5878

0

 

x

 

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

1.9

2

sin(2πx)

 

0.5878

0.9511

0.9511

0.5878

0

-0.5878

-0.9511

-0.9511

-0.5878

0

 Relationship between Variables

The relationship between the variables is not linear as shown in the following graph.


Task:

The task is to learn the equation that shows the relationship between the input(x) and output (2πx).

Polynomial Equation

So we can use a polynomial equation to represent the relationship.

Note : The shape of the polynomial is determined by the parameters.

Polynomial Approximations of sin(2πx) of Varying Degrees

p(x) = -4.1585x^3 + 4.1585x

p(x) = -4.9348x^5 + 9.8696x^3 - 4.9348x

p(x) = -28.2743x^9 + 126.0972x^7 - 252.1944x^5 + 201.7552x^3 - 67.2517x

Note that the co-efficients increase drastically as the degree increases.

Degree and Co-efficient of Polynomial

The co-efficient is learnt by the machine learning model. But assumptions have to be made on the degree of the polynomial.

Adding Noise to Dataset

If the exact equation is learnt while creating the machine learning model, the prediction on unseen data is low. So we can add noise to the data so that the machine learns the pattern ignoring the noise.

When degree = 0, 

the only line learnt is a line parallel to the x-axis


A polynomial of degree 0 is essentially a constant value (a horizontal line) that best fits the data in the least squares sense. This is equivalent to finding the mean of the y values in the dataset.

When degree = 1, 

any straight line can be learnt by the model. It does not understand even the training dataset and so it results in underfitting.

Fitting a polynomial of degree 1 (a linear polynomial) to your data involves finding the best linear relationship between your independent variable
���=��+��
w0 is the slope and
�
w1 is the y-intercept of the line.

When degree = 3, 

it identifies the pattern in the underlying data and this generalizes well the dataset. It can predict any example in the training set correctly. It can also predict well on unseen data.
General Form the Equation is 




When degree = 9, 

it learns the points in dataset and not the underlying pattern in the dataset. This results in overfitting. It can predict any example in the training set correctly. But is cannot predict well on unseen data.

Fitting a polynomial of degree 9 to your data is an advanced task, as it involves finding a high-degree polynomial that can closely follow the patterns (and potentially the noise) in your data.



Degree 25


Polynomial Curve Fitting with an ANN using pytorch can be with example in the following YouTube video.





Comments

Post a Comment

Popular posts from this blog

ANN Series - 10 - Backpropagation of Errors

-
Image
Watch the video to understand the forward pass in an ANN. Backpropagation, short for "backward propagation of errors," is a fundamental algorithm used for training artificial neural networks. It efficiently computes the gradient of the loss function with respect to the weights of the network, which is essential for adjusting the weights and minimizing the loss through gradient descent or other optimization techniques. The process involves two main phases: a forward pass and a backward pass.     Forward Pass   1. Input Layer:  The input features are fed into the network. 2. Hidden Layers:  Each neuron in these layers computes a weighted sum of its inputs (from the previous layer or the input layer) plus a bias term. This sum is then passed through an activation function to produce the neuron's output. This process repeats layer by layer until the output layer is reached. 3. Output Layer:  The final output of the network is computed, which is then used to calculate the loss
Read more

Regression 10 - Pseudo-Inverse Matrix Approach - Relationship between MLE and LSA - Derivation

-
Pseudo-Inverse Matrix Approach: The pseudo-inverse matrix approach is essentially the OLS approach. It provides a solution to the linear regression problem by minimizing the sum of squared residuals between the observed and predicted values. This approach is a direct algebraic method that does not make any probabilistic assumptions about the residuals; it simply finds the best fit in the least squares sense.   For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get from the pseudo-inverse matrix method, under the assumption of i.i.d. normal errors. In this blog we show how to prove that  Maximization of likelihood function under conditional Gaussian noise for a linear model is similar to the minimization of the sum of the squares error function.  The complete derivation along with formulae used are provided. Ensure that you go through the previous blogs to get a
Read more

Regression 9 - Maximum Likelihood Estimation(MLE) Approach for Regression - Derivation

-
  MLE Approach: When performing linear regression using MLE under the assumption that the residuals (errors) are independently and identically distributed (i.i.d.) with a normal distribution, we estimate the regression coefficients that maximize the likelihood of observing the given data. In this blog we see how to perform regression on a dataset that applies MLE for model fitting. The mathematical assumptions and derivations are given in detail.   The MLE approach gives you point estimates for the coefficients (mean of the likelihood distribution), and you can also compute the variance-covariance matrix of these estimates, which gives you the variances (and covariances) of the estimates. These variances are a measure of the uncertainty or the spread of the likelihood distribution of the parameter estimates. For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get
Read more