R² vs. Adjusted R²: Which Metric Tells the Real Story?
Understand the difference between R² and Adjusted R². Learn why R² always increases with more features, how Adjusted R² penalizes complexity, and when to use each metric for model comparison.
R² vs. Adjusted R²: Which Metric Tells the Real Story?
Understand how to evaluate your regression models accurately.
R² vs. Adjusted R²: Which Tells the Real Story?
When we build regression models to predict values, we need a way to measure how well they actually fit the data. Two of the most common metrics are R-squared (R²) and Adjusted R-squared. They sound similar, but they tell slightly different stories, especially when dealing with multiple input features!
Understanding the difference is crucial for correctly evaluating your models and avoiding the pitfall of thinking a complex model is great when it’s actually just overfitting.
R-Squared (R²): The Basic “Goodness-of-Fit”
What it Measures
R-squared tells you the proportion of the variance in your dependent variable (Y, what you’re predicting) that can be explained by the independent variable(s) (X, your inputs).
R² = 1 - (Sum of Squared Errors from Model) / (Total Sum of Squares)
R² = 1 - [ Σ(yᵢ - ŷᵢ)² / Σ(yᵢ - ȳ)² ]yᵢ= Actual valueŷᵢ= Predicted valueȳ= Mean of actual y values
Interpretation
- Values range from 0 to 1 (or 0% to 100%)
- R² = 0.80 means 80% of the variation in Y is explained by X
- Higher R² generally indicates better fit
- R² = 0 means the model explains none of the variability
The Big Limitation!
R² almost always increases whenever you add any new variable, even if it’s completely useless! This makes R² misleading when comparing models with different numbers of predictors.
Adjusted R-Squared: The Smarter Cousin
Why We Need It
Adjusted R² modifies R² to account for the number of independent variables (k) relative to sample size (n).
Adjusted R² = 1 - [(1 - R²) * (n - 1) / (n - k - 1)]n= Number of data pointsk= Number of predictors- The term
(n - 1) / (n - k - 1)acts as a penalty for added predictors
Key Benefit
- Only increases if adding a predictor genuinely improves the model beyond what’s expected by chance
- If you add a useless predictor, Adjusted R² will decrease
- Always ≤ R² (and can be negative if the model is very poor)
- Much better for comparing models with different numbers of predictors
R² vs. Adjusted R²: Side-by-Side
| Feature | R² | Adjusted R² |
|---|---|---|
| Effect of Adding Predictors | Always increases | Only increases if improvement is substantial |
| Best Use Case | Single model goodness-of-fit | Comparing models with different numbers of features |
| Overfitting Indicator | Can be misleadingly high | Helps detect overfitting |
| Range | 0 to 1 | Can be < 0, usually ≤ R² |
Which Metric Should You Use?
- R²: Report for your final model to show overall fit, but understand its limitation
- Adjusted R²: Use primarily when comparing models with varying numbers of variables
- Look at both! If R² is high but Adjusted R² is much lower, suspect overfitting or irrelevant features
Red Flag: Large Gap Between R² and Adjusted R²
If R² = 0.90 but Adjusted R² = 0.75, your model likely contains irrelevant features or is overfitting.
R² vs. Adjusted R²: Key Takeaways
- R² measures % of variance in Y explained by X(s) — ranges 0-1, higher is better
- Limitation of R²: Increases with any predictor added, even useless ones
- Adjusted R² accounts for number of predictors — penalizes complexity
- Adjusted R² only increases if the added predictor significantly improves the model
- For model comparison: Use Adjusted R² when comparing models with different feature counts
- Large gap warning: A large gap between R² and Adjusted R² signals potential overfitting