Overfitting and Regularization: L1 & L2 Explained — ML Breadth

Controlling Model Complexity

Core Concepts to Master

Overfitting: The core problem where a model learns the training data too well, including its noise, and fails to generalize to new data.
Model Complexity: The idea that more complex models (e.g., higher-degree polynomials, deep decision trees) are more prone to overfitting.
Loss Function + Penalty: The fundamental mechanism of regularization—modifying the model's objective to penalize complexity (large coefficient values).
L1 vs. L2 Norm: The mathematical difference between the penalty terms (sum of absolute values vs. sum of squared values).
Sparsity: The unique ability of L1 regularization to shrink some feature coefficients to exactly zero, performing automatic feature selection.
Geometric Interpretation: Visualizing the interaction between the loss function contours and the constraint regions (diamond vs. circle) to understand why L1 produces sparse solutions.

Interview Walkthrough

Interviewer: Let's talk about a core concept in machine learning. What causes overfitting, and how do L1 and L2 regularization help prevent it? I'm particularly interested in the geometric interpretation of both.

Candidate: Of course. Overfitting is a fundamental challenge where a model learns the training data too well, to the point that it captures not just the underlying signal but also the random noise. This leads to excellent performance on the training data but poor performance on new, unseen data.

What Causes Overfitting?

Think of it as the difference between learning and memorizing. A good student learns the concepts to pass an exam. A bad student memorizes the answers to the practice questions. The bad student will fail when the exam has new questions.

Technically, overfitting is caused by a model that has too much complexity for the amount of data available. A highly complex model (like a high-degree polynomial) has enough flexibility to wiggle and bend its decision boundary to perfectly fit every single data point, including the outliers and noise.

The Overfitting Problem

Show Overfit Model

How Regularization Helps

Regularization is a technique to prevent overfitting by discouraging model complexity. It does this by adding a penalty term to the model's loss function. The new objective becomes:

New Loss = Original Loss (e.g., MSE) + λ * Penalty Term

Where `λ` (lambda) is a hyperparameter that controls the strength of the penalty. Now, the model must find a balance: it wants to minimize the original loss (fit the data) but also minimize the penalty (keep itself simple). This forces the model to learn only the most important patterns.

L1 vs. L2 Regularization

L1 and L2 are just two different ways of defining that penalty term.

L1 Regularization (LASSO)

Penalty: The sum of the absolute values of the coefficients. `λ * Σ|βᵢ|`
Effect: L1 has a unique property: it can shrink the coefficients of less important features to exactly zero. This means it effectively performs automatic feature selection, resulting in a sparse model.

L2 Regularization (Ridge)

Penalty: The sum of the squared values of the coefficients. `λ * Σ(βᵢ)²`
Effect: L2 shrinks all coefficients towards zero but never to exactly zero. It reduces the influence of all features, a process often called "weight decay," but it doesn't eliminate any.

The Geometric Interpretation

This is the best way to understand why their effects are different. Imagine a 2D space where the axes are two model weights, `β₁` and `β₂`.

The elliptical contours represent the model's loss function. The center of the ellipses is the point of minimum loss (the unconstrained solution, like OLS).
The shaded region represents the "penalty budget" allowed by the regularization term. The model must find a solution inside this region.

L1 (LASSO) - Diamond Constraint

The L1 constraint is a diamond. The loss ellipses are likely to make first contact with the diamond at one of its corners, which lie on the axes. When the solution is on an axis, one of the coefficients is exactly zero. This is why L1 produces sparse models.

L2 (Ridge) - Circle Constraint

The L2 constraint is a circle. Since a circle has no sharp corners, the point of tangency between the ellipse and the circle will almost never be exactly on an axis. Therefore, the coefficients are shrunk towards zero but do not become zero.

Interviewer: That's a perfect geometric explanation. So, given their different effects, when would you choose to use L1 over L2, and vice versa?

Candidate: The choice depends directly on the assumptions about the features and the desired properties of the final model.

When to Choose L1 (LASSO):

For Feature Selection: If you suspect that many of your features are irrelevant or redundant, L1 is the ideal choice because its sparsity property will automatically perform feature selection for you.
For Interpretability: When you need a simpler, more interpretable model. A model with fewer non-zero coefficients is easier to explain to stakeholders.

When to Choose L2 (Ridge):

When You Believe All Features are Relevant: If you think most features have at least some predictive value, L2 is better because it reduces their impact without eliminating them entirely.
To Handle Collinearity: L2 is more effective at handling multicollinearity (highly correlated features). It tends to distribute the weight among correlated features, while L1 might arbitrarily pick one and set the others to zero.
As a General Default: L2 is often a safer default choice for regularization because it's less "aggressive" than L1 and provides more stable solutions.

In practice, the choice can also be determined empirically through cross-validation to see which penalty leads to a better generalization score on unseen data.

Why This Comparison Matters in an Interview

Shows Deep Understanding of Overfitting: A candidate who can explain regularization demonstrates a mature understanding of how to combat one of the most common problems in machine learning.
Connects Math to Practical Outcomes: Linking the L1 norm to sparsity and the L2 norm to weight decay shows you understand the *consequences* of the mathematical formulas.
Strong Communication Skills: The geometric interpretation is a classic test of a candidate's ability to explain complex mathematical concepts in a simple, visual way.
Demonstrates Practical Decision-Making: Knowing when to prefer L1 (for sparsity) versus L2 (for collinearity/stability) shows you can make informed modeling choices based on data characteristics.

Pro-Tip: To show even more advanced knowledge, mention Elastic Net regularization. Explain that it's a hybrid of L1 and L2, combining both penalties. This allows it to perform feature selection like L1 while retaining the stability of L2 in the presence of highly correlated features, often providing the best of both worlds.

Which Regularization is Best?

For each scenario, choose the most appropriate regularization strategy.

Scenario 1: Feature Selection

You have 200 features but believe only about 15-20 are actually useful. You want the model to automatically discard the irrelevant ones. Which method is designed for this?

Scenario 2: Correlated Features

Your dataset has highly correlated features like 'height_cm' and 'height_inches'. You want to keep both but reduce their combined influence. Which method is more stable for this?

Scenario 3: Tuning Lambda (λ)

You apply L2 regularization with a very large `λ` value. What will be the effect on the model's coefficients?