Beyond Straight Lines: Understanding Polynomial Regression

How Does it Create Curves?

Adding Powers of X

Remember the simple linear equation: y = b₀ + b₁x. This can only draw straight lines.

Polynomial Regression extends this by adding higher powers of the independent variable x:

By adding terms like x² (which creates a parabola), x³ (which creates an S-shape), etc., the model can create much more flexible curves that can fit non-linear data better.

Think of it like this: a degree 1 polynomial is a straight ruler. A degree 2 polynomial is like bending the ruler once. A degree 3 polynomial is like bending it twice, and so on. More bends allow it to fit more complex shapes.

It's Still "Linear"?

Interestingly, even though the *relationship* between X and Y is curved (non-linear), it's still considered a type of linear model in a statistical sense. Why? Because the equation is linear *with respect to the coefficients* (b₀, b₁, b₂, ...). We are still just finding the best weights for each term, even if those terms involve powers of X. This means we can still use the same `LinearRegression` techniques from libraries like Scikit-learn to solve for the coefficients!

How to Implement Polynomial Regression

The key trick is to transform our original independent variable(s) into polynomial features before fitting a standard linear regression model.

1. Load & Prepare Data:
   • Import your data (e.g., using `pandas`).
   • Separate your independent variable(s) X and dependent variable y.
   • Perform basic preprocessing like handling missing values if necessary. (In the `Position_Salaries` example, this might not be needed).
2. Create Polynomial Features:
   • This is the crucial step! Use Scikit-learn's PolynomialFeatures class.
   • You specify the desired degree (e.g., `degree=2`, `degree=3`, `degree=4`).
   • Use the fit_transform method on your original X data. This will generate a new array X_poly containing the original column(s), plus columns for x², x³, etc., up to the specified degree.
   • PolynomialFeatures automatically adds a column of ones for the intercept term.
3. Train a Linear Regression Model:
   • Create an instance of Scikit-learn's LinearRegression.
   • Fit this linear model using the polynomial features (X_poly) and the original target variable (y):
     model.fit(X_poly, y).
4. Make Predictions:
   • To predict for a new value of X, you first need to transform that new X using the *same* `PolynomialFeatures` object (`poly_reg.transform(new_X)`).
   • Then, use the trained linear model to predict on these transformed features: `model.predict(transformed_new_X)`.
5. Evaluate & Visualize:
   • Evaluate using metrics like MSE or R² (comparing predictions on transformed test data vs actual test data).
   • Visualize by plotting the original data points and the predicted curve from the polynomial model.

Python Code Example

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

# 1. Load dataset
dataset = pd.read_csv('Position_Salaries.csv')
X = dataset.iloc[:, 1:2].values # Level (Independent) - keep as 2D array
y = dataset.iloc[:, -1].values  # Salary (Dependent)

# (Optional) Fit Linear Regression for comparison
lin_reg = LinearRegression()
lin_reg.fit(X, y)

# 2. Create Polynomial Features (e.g., degree 4)
poly_reg = PolynomialFeatures(degree=4) # Choose the degree
X_poly = poly_reg.fit_transform(X) # Transform X into X, X^2, X^3, X^4

# Print X_poly to see the transformation (includes column of 1s for intercept)
print("Original X:\n", X[:3])
print("\nPolynomial Features (X_poly, degree 4):\n", X_poly[:3])

# 3. Train Linear Regression on Polynomial Features
poly_lin_reg = LinearRegression()
poly_lin_reg.fit(X_poly, y) # Fit model using the transformed features

# 4. Visualize Results

# --- Visualize Linear Fit ---
plt.figure(figsize=(10, 6))
plt.scatter(X, y, color='#ec4899', label='Actual Salary') # Pink dots
plt.plot(X, lin_reg.predict(X), color='#3b82f6', label='Linear Fit') # Blue line
plt.title('Salary vs Level (Linear Regression)')
plt.xlabel('Position Level')
plt.ylabel('Salary')
plt.legend()
# plt.show()

# --- Visualize Polynomial Fit (smoother curve) ---
# Create smoother X values for plotting the curve
X_grid = np.arange(min(X), max(X), 0.1)
X_grid = X_grid.reshape((len(X_grid), 1))
# Transform the smooth X values using the SAME poly_reg object
X_grid_poly = poly_reg.transform(X_grid)

plt.figure(figsize=(10, 6))
plt.scatter(X, y, color='#ec4899', label='Actual Salary') # Pink dots
plt.plot(X_grid, poly_lin_reg.predict(X_grid_poly), color='#14b8a6', label=f'Polynomial Fit (Degree 4)') # Teal curve
plt.title('Salary vs Level (Polynomial Regression)')
plt.xlabel('Position Level')
plt.ylabel('Salary')
plt.legend()
# plt.show()

# 5. Predicting a new result (e.g., for level 6.5)
new_level = [[6.5]]
# Transform the new level using the fitted poly_reg
new_level_poly = poly_reg.transform(new_level)
predicted_salary = poly_lin_reg.predict(new_level_poly)
print(f"\nPredicted salary for Level 6.5: ${predicted_salary[0]:,.2f}")
                                    

Choosing the Right Degree: A Balancing Act

A key decision in Polynomial Regression is choosing the degree of the polynomial.

• Too Low Degree (e.g., 1): If the data is truly curved, a low degree will act like linear regression and underfit (high bias). It won't capture the pattern.
• Too High Degree (e.g., 10): The curve can become extremely wiggly and fit the training data *perfectly*, including the noise. This model will overfit (high variance) and perform poorly on new data it hasn't seen before.
• Just Right Degree: This captures the underlying trend without fitting the noise, balancing bias and variance.

How do you find the right degree?

• Visualization: Plot fits for different degrees and see which one looks most reasonable.
• Evaluation Metrics: For larger datasets where you have a test set, check metrics like MSE or R² on the test set for different degrees. Choose the degree that gives the best performance on *unseen* data.
• Cross-Validation: A more robust technique for larger datasets to find the optimal degree that generalizes well.

Generally, start with degree 2, then try 3 or 4, and compare results. Very high degrees are often a sign of overfitting.

Key Takeaways: Polynomial Regression

• Used when the relationship between one input (X) and one output (Y) is curved (non-linear).
• Works by adding powers of X (like X², X³, etc.) to the linear equation.
• The highest power used is the degree of the polynomial.
• Implemented by creating polynomial features from X using PolynomialFeatures and then fitting a standard LinearRegression model to these new features.
• Choosing the right degree is crucial to avoid underfitting (too simple) or overfitting (too complex).
• Visualize the fit to help choose the degree and assess performance.