Regression1. Regression Overview with Python Code

-

A detailed explanation for Regression can be viewed in the following YouTube video.


Regression vs Classification

A regression task in the context of machine learning and statistics is a type of problem where the goal is to predict a continuous outcome variable based on one or more predictor variables. It's different from classification tasks, where the goal is to predict a discrete label. Regression is used to understand relationships between variables and for predicting trends or future values.

The output to a regression problem is a vector of real numbers.

 Understanding Regression

1. Continuous Outcome Variable: 

This is a variable that can take any value within a range. It can take decimal values and not just integer values. They are the dependent variables.

Example

For instance, predicting temperatures, prices, or distances.

The distance is a real number say 32km. Even though it looks like an integer, it may actually be a real number say 32.0000004 subsuming km and some m and some cm.

2. Predictor Variables: 

Also known as independent variables, these are the inputs or factors that you suspect have an impact on the outcome variable. 

Example

For example, in predicting house prices, predictor variables might include the size of the house, its age, and the number of bedrooms.

3. Modeling the Relationship: 

Regression involves finding a mathematical equation that describes the relationship between the predictor and outcome variables. 

Example

The simplest form of regression is linear regression, which assumes a straight-line relationship.

 Types of Regression

1. Simple Linear Regression: 

Involves one predictor and one outcome variable. The relationship is modeled with a straight line: 

y = mx + b

where y is the outcome, x is the predictor, m is the slope of the line, and b is the y-intercept. They are linear in the co-efficient and x.

 2. Multiple Linear Regression: 

Uses more than one predictor variable. The model looks like: 

b0 - intercept

coefficients : 

x1, x2, ... xn : Predictor Variables
They are linear in the co-efficient and x.

3. Non-Linear Regression: 

For more complex relationships that can't be captured with a straight line, models like polynomial regression or logistic regression (for specific kinds of non-linear relationships) are used.

The polynomial model looks like

The logistic function looks like:


The polynomial model is still linear in the parameters. It is non-linear x.

 Performing a Regression Task

1. Data Collection: 

Gather data on the variables of interest. In our example, we have 2 variable of interest namely x and t.

import numpy as np
import matplotlib.pyplot as plt

# Define the number of samples
num_samples = 100

# Create an array of 100 points between 0 and 1
x = np.linspace(0, 1, num_samples)

# Compute the sine of 2*pi*x
y = np.sin(2 * np.pi * x)

# Add Gaussian noise to y
# Set a mean and a standard deviation for the Gaussian noise
noise_mean = 0
noise_std_dev = 0.1

# Generate Gaussian noise
noise = np.random.normal(noise_mean, noise_std_dev, y.shape)

# Create a new y dataset with noise added
y_noisy = y + noise

# Combine x and y into a single dataset for the original and noisy data
dataset_original = np.column_stack((x, y))
dataset_noisy = np.column_stack((x, y_noisy))

# Plot the original dataset
plt.scatter(x, y, label='Original Dataset')

# Plot the noisy dataset
plt.scatter(x, y_noisy, color='red', label='Noisy Dataset', alpha=0.6)

# Label the axes
plt.xlabel('x')
plt.ylabel('y')

# Title of the plot
plt.title('Original vs Noisy Data')

# Show legend
plt.legend()

# Show grid
plt.grid(True)

# Show the plot on the screen
plt.show()

 2. Exploratory Analysis: 

Understand the data, check for correlations, and prepare it for modeling (handling missing values, encoding categorical variables, etc.).

 3. Model Selection: 

Choose an appropriate regression model based on the nature of the data and the relationship between variables. 

 4. Training the Model: 

Use the collected data to train the model. This involves finding the values of the parameters (like m and b in linear regression) that best fit the data.

import numpy as np
import matplotlib.pyplot as plt

# Assuming x, y, and y_noisy are already defined as in previous steps

# Fit a polynomial of degree 3 (cubic) to the noisy data
coefficients = np.polyfit(x, y_noisy, 3)

# Print the coefficients
print("Coefficients of the polynomial:", coefficients)

# Create a polynomial function using the fitted coefficients
p = np.poly1d(coefficients)

# Generate y values using the polynomial function
y_fit = p(x)

 5. Evaluation: 

Assess the model's performance using metrics like R-squared, Mean Squared Error (MSE), or Root Mean Squared Error (RMSE).

from sklearn.metrics import r2_score, mean_squared_error

# Calculate and print the R² value
r_squared = r2_score(y, y_fit)
print("R² value:", r_squared)


# Calculate MSE and RMSE
mse = mean_squared_error(y, y_fit)
rmse = np.sqrt(mse)
print("MSE:", mse)
print("RMSE:", rmse)

 6. Prediction: 

Use the model to make predictions on new, unseen data.

    import numpy as np


# Two given x values to predict
x_predict1 = 0.523  # Example value, replace with your value
x_predict2 = 0.101 # Example value, replace with your value

# Predicting y values for the given x values
y_predict1 = p(x_predict1)
y_predict2 = p(x_predict2)

# Print the predictions
print("Prediction for x =", x_predict1, "is y =", y_predict1)
print("Prediction for x =", x_predict2, "is y =", y_predict2)

 7. Refinement: 

Based on the model's performance, it might be necessary to return to previous steps, select a different model, or gather more data.

Github Link for Python Code:

The following code helps to understand the concept of regression.

ml-course/polynomial.ipynb at main · lovelynrose/ml-course (github.com)

The following code helps to understand all the steps in performing regression. 

ml-course/regression_basics_with_California_dataset.ipynb at main · lovelynrose/ml-course (github.com)

 Applications

Regression tasks are ubiquitous in real-world scenarios like predicting sales in business, estimating housing prices in real estate, forecasting weather conditions in meteorology, and many others. The key is that the outcome variable we want to predict or understand varies continuously and we believe this variation can be explained by other variables.

Comments

Post a Comment

Popular posts from this blog

ANN Series - 10 - Backpropagation of Errors

-
Image
Watch the video to understand the forward pass in an ANN. Backpropagation, short for "backward propagation of errors," is a fundamental algorithm used for training artificial neural networks. It efficiently computes the gradient of the loss function with respect to the weights of the network, which is essential for adjusting the weights and minimizing the loss through gradient descent or other optimization techniques. The process involves two main phases: a forward pass and a backward pass.     Forward Pass   1. Input Layer:  The input features are fed into the network. 2. Hidden Layers:  Each neuron in these layers computes a weighted sum of its inputs (from the previous layer or the input layer) plus a bias term. This sum is then passed through an activation function to produce the neuron's output. This process repeats layer by layer until the output layer is reached. 3. Output Layer:  The final output of the network is computed, which is then used to calculate the loss
Read more

Regression 10 - Pseudo-Inverse Matrix Approach - Relationship between MLE and LSA - Derivation

-
Pseudo-Inverse Matrix Approach: The pseudo-inverse matrix approach is essentially the OLS approach. It provides a solution to the linear regression problem by minimizing the sum of squared residuals between the observed and predicted values. This approach is a direct algebraic method that does not make any probabilistic assumptions about the residuals; it simply finds the best fit in the least squares sense.   For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get from the pseudo-inverse matrix method, under the assumption of i.i.d. normal errors. In this blog we show how to prove that  Maximization of likelihood function under conditional Gaussian noise for a linear model is similar to the minimization of the sum of the squares error function.  The complete derivation along with formulae used are provided. Ensure that you go through the previous blogs to get a
Read more

Regression 9 - Maximum Likelihood Estimation(MLE) Approach for Regression - Derivation

-
  MLE Approach: When performing linear regression using MLE under the assumption that the residuals (errors) are independently and identically distributed (i.i.d.) with a normal distribution, we estimate the regression coefficients that maximize the likelihood of observing the given data. In this blog we see how to perform regression on a dataset that applies MLE for model fitting. The mathematical assumptions and derivations are given in detail.   The MLE approach gives you point estimates for the coefficients (mean of the likelihood distribution), and you can also compute the variance-covariance matrix of these estimates, which gives you the variances (and covariances) of the estimates. These variances are a measure of the uncertainty or the spread of the likelihood distribution of the parameter estimates. For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get
Read more