Logistic Regression Explained: Predicting Categories

Why Not Use Linear Regression for Classification?

You might wonder, "Can't we just use the straight line from Linear Regression?" For classification, usually not. Here's why:

• Output Isn't Probability: Linear Regression predicts continuous numbers that can go below 0 or above 1. For classification, we want a probability between 0 and 1 (the chance of belonging to a specific class).
• Sensitivity to Outliers: Linear Regression lines can be heavily influenced by outliers, potentially shifting the decision point incorrectly.

We need a way to take the output of a linear-like equation and squash it neatly into the 0-to-1 probability range. That's where the magic happens!

Making the Decision: The Boundary

From Probability to Class

The model outputs a probability (e.g., 0.7, 0.2, 0.5). But usually, we need a definite class label (e.g., 'Yes' or 'No', 1 or 0). How do we decide?

We use a Decision Boundary (or threshold). The most common threshold is 0.5:

• If the predicted probability σ(z) is ≥ 0.5, we classify the instance as Class 1 (Positive).
• If the predicted probability σ(z) is < 0.5, we classify the instance as Class 0 (Negative).

This threshold corresponds to the point where the linear part z = b₀ + b₁x₁ + ... equals zero. In geometric terms, this often creates a linear boundary (a line, plane, or hyperplane) separating the classes in the feature space.

Image Credit: Bayesian via StackExchange / Wikipedia, CC BY-SA 4.0

Adjusting the Threshold

While 0.5 is common, you can adjust this threshold depending on your specific needs:

• In medical diagnosis (like cancer detection), you might lower the threshold (e.g., to 0.3). This makes the model more likely to predict 'Cancer' (Class 1), increasing Recall (finding more true cases) but potentially increasing False Positives. You prioritize not missing actual cases.
• In spam filtering, you might raise the threshold (e.g., to 0.8). This makes the model more confident before marking an email as 'Spam' (Class 1), increasing Precision (fewer important emails marked as spam) but potentially increasing False Negatives (letting more spam through).

Building a Logistic Regression Model (Python/Sklearn)

Here’s a standard workflow:

1. Load & Prepare Data: Import data using Pandas. Handle any missing values. Separate features (X) and the target variable (y). Ensure 'y' contains your categorical labels (e.g., 0 and 1).
2. Split Data: Divide into training and testing sets using `train_test_split`.
3. Feature Scaling: Very important for Logistic Regression, especially if regularization is used or if features have different scales. Use `StandardScaler` to scale X_train and X_test. Fit the scaler ONLY on X_train.
4. Train the Model:
   • Import `LogisticRegression` from `sklearn.linear_model`.
   • Create an instance: `model = LogisticRegression(random_state=0)` (setting `random_state` ensures reproducibility).
   • Fit the model to the scaled training data: `model.fit(X_train_scaled, y_train)`.
5. Make Predictions: Predict probabilities (`predict_proba`) or class labels (`predict`) on the scaled test data (`X_test_scaled`).
6. Evaluate the Model: Assess performance using metrics appropriate for classification:
   • Confusion Matrix: Shows TP, TN, FP, FN. Use `confusion_matrix(y_test, y_pred)`.
   • Accuracy Score: Overall percentage correct. Use `accuracy_score(y_test, y_pred)`.
   • Precision, Recall, F1-Score: Especially important for imbalanced data. Use `classification_report(y_test, y_pred)`.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, accuracy_score, classification_report

# --- Assume X, y are loaded and preprocessed (missing values handled) ---

# 1. Split Data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# 2. Feature Scaling
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# 3. Train Model
log_reg_model = LogisticRegression(random_state=42)
log_reg_model.fit(X_train_scaled, y_train)

# 4. Predict on Test Set
y_pred = log_reg_model.predict(X_test_scaled)

# 5. Evaluate
cm = confusion_matrix(y_test, y_pred)
acc = accuracy_score(y_test, y_pred)
report = classification_report(y_test, y_pred)

print("Confusion Matrix:\n", cm)
print(f"\nAccuracy Score: {acc:.4f}")
print("\nClassification Report:\n", report)