Logistic Regression

-

 Logistic regression is a statistical method for binary classification. The classes can be considered as Y={0,1}. For a dataset X, we are trying to find

P(Y=1|X) 

Logistic Regression Workflow

Linear Combination(logit)

The logit is calculated first as shown in the blog on Softmax. This is a linear combination of the input features given by

logit = t = w0 + w1 x1 + w2 x2 + ... + wn xn

This can be a value between (-INF, +INF). This value also reflects the log odds of P(Y=1|X).

Odds

The odds of an event occurring is the ratio between the probability of the event occurring to the probability of the event not occurring.

Odds = {P(Y=1|X} / {1 - P(Y=1|X}

Log Odds

This is the natural logarithm of the odds. In logistic regression we see that the log odds of P(Y=1|X) is the logit(t). The proof can be found in the pdf below.

Sigmoid Transformation

The logit(t) is passed through a sigmoid function to get a value between 0 and 1 and is interpreted as the probability of getting class Y=1. i.e. P(Y=1|X). In other words, the logistic function converts the log odds into a probability.

The following pdf gives the mathematical formulation of the logistic regression.


Interpretation of the coefficients in terms of odds ratios(AI Generated Portion)

The interpretation of the coefficients in terms of odds ratios is what makes logistic regression particularly insightful for understanding how various predictors influence the likelihood of different outcomes.

1. Positive Coefficient (t>0): 

When a coefficient is positive, its exponentiated form (e^t) will be greater than 1. This means that for every one-unit increase in the predictor variable, the odds of the outcome occurring (assuming the outcome is represented as (Y=1)) are multiplied by e^t, indicating a positive association between the predictor and the outcome. The greater the value of e^t, the stronger the positive association.

2. Negative Coefficient (t<0): 

If a coefficient is negative, e^t will be less than 1. This indicates that for every one-unit increase in the predictor variable, the odds of the outcome are multiplied by a value less than 1, effectively decreasing. Hence, there's a negative association between the predictor and the outcome; the predictor's increase is associated with a decrease in the odds of the outcome.

 3. Zero Coefficient (t= 0): 

When a coefficient is zero, e^t equals 1. This scenario means that the predictor has no effect on the odds of the outcome; there's no association between the predictor variable and the likelihood of the outcome occurring.

  Example for Clarity:

 - If a logistic regression model yields a coefficient t of 0.5 for a predictor (say, age), then (e^{0.5} approx 1.65). This means that with each additional year of age, the odds of the target event (e.g., purchasing a specific product) increase by 65%, holding all other variables constant.

 - Conversely, if another predictor (say, smoking) has a coefficient (t) of -0.3, then (e^{-0.3} approx 0.74). Thus, for each additional unit increase in smoking (however that's quantified), the odds of the target event decrease by 26%, all else being equal.

 - If a predictor (like the presence of a certain feature in a product) has a coefficient (t) of 0, its odds ratio (e^{0} = 1), indicating it has no effect on the likelihood of the outcome.

This interpretation of the coefficients in terms of odds ratios is what makes logistic regression particularly insightful for understanding how various predictors influence the likelihood of different outcomes.

Comments

Post a Comment

Popular posts from this blog

ANN Series - 10 - Backpropagation of Errors

-
Image
Watch the video to understand the forward pass in an ANN. Backpropagation, short for "backward propagation of errors," is a fundamental algorithm used for training artificial neural networks. It efficiently computes the gradient of the loss function with respect to the weights of the network, which is essential for adjusting the weights and minimizing the loss through gradient descent or other optimization techniques. The process involves two main phases: a forward pass and a backward pass.     Forward Pass   1. Input Layer:  The input features are fed into the network. 2. Hidden Layers:  Each neuron in these layers computes a weighted sum of its inputs (from the previous layer or the input layer) plus a bias term. This sum is then passed through an activation function to produce the neuron's output. This process repeats layer by layer until the output layer is reached. 3. Output Layer:  The final output of the network is computed, which is then used to calculate the loss
Read more

Regression 10 - Pseudo-Inverse Matrix Approach - Relationship between MLE and LSA - Derivation

-
Pseudo-Inverse Matrix Approach: The pseudo-inverse matrix approach is essentially the OLS approach. It provides a solution to the linear regression problem by minimizing the sum of squared residuals between the observed and predicted values. This approach is a direct algebraic method that does not make any probabilistic assumptions about the residuals; it simply finds the best fit in the least squares sense.   For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get from the pseudo-inverse matrix method, under the assumption of i.i.d. normal errors. In this blog we show how to prove that  Maximization of likelihood function under conditional Gaussian noise for a linear model is similar to the minimization of the sum of the squares error function.  The complete derivation along with formulae used are provided. Ensure that you go through the previous blogs to get a
Read more

Regression 9 - Maximum Likelihood Estimation(MLE) Approach for Regression - Derivation

-
  MLE Approach: When performing linear regression using MLE under the assumption that the residuals (errors) are independently and identically distributed (i.i.d.) with a normal distribution, we estimate the regression coefficients that maximize the likelihood of observing the given data. In this blog we see how to perform regression on a dataset that applies MLE for model fitting. The mathematical assumptions and derivations are given in detail.   The MLE approach gives you point estimates for the coefficients (mean of the likelihood distribution), and you can also compute the variance-covariance matrix of these estimates, which gives you the variances (and covariances) of the estimates. These variances are a measure of the uncertainty or the spread of the likelihood distribution of the parameter estimates. For simple linear regression, the MLE estimates of the coefficients will actually be the same as the Ordinary Least Squares (OLS) estimates, which are also the same as what you get
Read more