Regression 5. Least Squares Approach for Linear Regression - Derivation

-

Ordinary Least Squares (OLS) is a fundamental method in statistical modeling, particularly for linear regression. The goal of OLS is to find the best-fitting line through a set of data points that minimizes the sum of the squared differences (residuals) between the observed values and the values predicted by the model.

Mathematical Formulation of OLS with and without Covariance Matrix

Comments

Post a Comment

Popular posts from this blog

ANN Series - 10 - Backpropagation of Errors

-
Image
Watch the video to understand the forward pass in an ANN. Backpropagation, short for "backward propagation of errors," is a fundamental algorithm used for training artificial neural networks. It efficiently computes the gradient of the loss function with respect to the weights of the network, which is essential for adjusting the weights and minimizing the loss through gradient descent or other optimization techniques. The process involves two main phases: a forward pass and a backward pass.     Forward Pass   1. Input Layer:  The input features are fed into the network. 2. Hidden Layers:  Each neuron in these layers computes a weighted sum of its inputs (from the previous layer or the input layer) plus a bias term. This sum is then passed through an activation function to produce the neuron's output. This process repeats layer by layer until the output layer is reached. 3. Output Layer:  The final output of the network is computed, which is then used to cal...
Read more

ANN Series 11 - Regularization in Neural Networks

-
Image
Overfitting Regularization in neural networks is a crucial technique used to prevent overfitting , which occurs when a model learns the training data too well, including its noise and outliers, leading to poor performance on unseen data. Overfitting happens especially when the network is too complex relative to the amount and variety of the training data.  Regularization techniques modify the learning process to reduce the complexity of the model , encouraging it to learn more general patterns that can generalize better to new, unseen data.  Common Techniques Here are some common regularization techniques used in neural networks: 1. L1 Regularization (Lasso Regression):  Adds a penalty equal to the absolute value of the magnitude of coefficients. This can lead to sparse models where some weights become exactly zero, effectively removing some features/weights. Lasso can struggle in situations where the number of predictors is much larger than the number of observation...
Read more

Softmax Function - First Derivative

-
 Logits They refer to the raw output produced by a machine learning model. This is before we normalize the data to the expected form. For Example, consider an ANN for classifying into 3 classes. The output layer may have 3 output neurons with a linear activation function. This is a logits. Note this will be a vector of 3 real values.  Softmax function This function is used when we want to interpret the output of a model as the probability for various classes. This is specifically useful in a multi-class classification problem. It is a squashing function that squashes the values to fall between [0,1] and sum to 1 across all classes. It evaluates the probability of choosing a class by using the logit across all the classes. The final predicted class has the highest probability. This is commonly used in ANN as the last layer of a multi-class classification problem. The node with the highest probability represents the chosen class. Probability Distribution A probability distributi...
Read more