#Lecture 009 — Overfitting, Underfitting & Regularization

Lecture 008 gave us metrics to measure a model's quality. This lecture explains the single most common reason those metrics look bad — and the fix that comes up constantly from here through neural networks in Lecture 010.

Contents

Prerequisites · Connection · Objectives · Lesson · Code Examples · Common Mistakes · Best Practices · Mini Challenge · Practice Questions · Answers · Takeaways

#Prerequisites

• Lecture 008 — train/test evaluation and cross-validation
• Lecture 005 — linear regression's underfitting pitfall, referenced again here

#Connection With Previous Lectures

Lecture 005 briefly flagged that a straight line underfits curved data; this lecture names that pattern properly, pairs it with its opposite (overfitting), and gives the standard fix (regularization) — directly relevant background for tuning any model from Lectures 005–007.

#Learning Objectives

By the end of this lecture you will be able to:

• Distinguish overfitting from underfitting using training vs. test performance
• Explain the bias-variance tradeoff in one sentence
• Name the fixes for each problem
• Explain what L1 and L2 regularization penalize, and how they differ

#Lesson

Underfitting happens when a model is too simple to capture the real pattern — it performs poorly on both training and test data (high bias, low variance). Overfitting happens when a model learns the training data too well, including its noise — it performs great on training data but poorly on new data (low bias, high variance). This is the core bias-variance tradeoff: more model complexity reduces bias but raises variance, and vice versa; the goal is the sweet spot that minimizes both.

Fixes for underfitting: use a more complex model, add/engineer features, reduce regularization, train longer, scale features properly.

Fixes for overfitting: collect more data, reduce model complexity, apply regularization, use dropout (for neural nets — Lecture 010), use early stopping, clean noisy data.

#Regularization

Regularization adds a penalty for model complexity to the loss function:

• L1 / Lasso: penalty = λ·Σ|wⱼ| — can shrink some coefficients to exactly zero, performing automatic feature selection.
• L2 / Ridge: penalty = λ·Σwⱼ² — shrinks correlated coefficients together, helping with multicollinearity, but rarely zeroes them out.
• Elastic Net: a weighted mix of L1 and L2 (mixing parameter α; α=1 is pure Lasso, α=0 is pure Ridge).

#Code Examples

#Example 1 — Comparing Lasso, Ridge, and Elastic Net

from sklearn.linear_model import Lasso, Ridge, ElasticNet
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X, y = make_regression(n_samples=100, n_features=5, noise=0.1, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

for name, model in [("Lasso", Lasso(alpha=0.1)), ("Ridge", Ridge(alpha=1.0)),
                     ("ElasticNet", ElasticNet(alpha=1.0, l1_ratio=0.5))]:
    model.fit(X_train, y_train)
    mse = mean_squared_error(y_test, model.predict(X_test))
    print(name, "MSE:", mse)

alpha controls how strongly the penalty is applied — higher alpha means more regularization, pushing coefficients closer to (or, for Lasso, exactly to) zero.

#Common Beginner Mistakes

1. Judging a model only by training accuracy/loss — a model with 99% training accuracy and 60% test accuracy isn't "almost perfect," it's badly overfit. Always evaluate on held-out data (Lecture 008), and get suspicious of a training score that's "too good."

2. Reaching for a more complex model as the default fix for poor performance — if the model is already overfitting, adding complexity makes it worse; check which side of the bias-variance tradeoff the problem is actually on first.

3. Applying heavy regularization blindly — too much regularization turns an overfitting problem into an underfitting one; alpha needs tuning (via cross-validation), not guessing.

#Best Practices

• Always compare training and test/validation performance together, never training performance alone.
• Diagnose before fixing: a large gap between train and test score means overfitting; both being poor means underfitting.
• Tune regularization strength (alpha) with cross-validation rather than a fixed guess.

#Mini Challenge

Run Example 1 with alpha set much higher (e.g., 10.0) for Ridge and compare the MSE — does performance get better or worse, and why?

#Practice Questions

1. How do you tell overfitting apart from underfitting using training and test scores?
2. What does the bias-variance tradeoff describe in one sentence?
3. What's the practical difference between L1 and L2 regularization's effect on coefficients?
4. Name two fixes for overfitting that don't involve regularization.

#Practice Answers

1. Underfitting: poor performance on both training and test data. Overfitting: strong training performance but much weaker test performance.
2. More model complexity reduces bias but increases variance, and vice versa — the best model finds the sweet spot that minimizes both.
3. L1 (Lasso) can shrink some coefficients to exactly zero, effectively selecting features; L2 (Ridge) shrinks correlated coefficients together but rarely zeroes them out.
4. Any two of: collect more data, reduce model complexity, use dropout, use early stopping, clean noisy data.

#Key Takeaways

• Underfitting = poor on both train and test; overfitting = great on train, poor on test
• The bias-variance tradeoff is the reason more complexity isn't always better
• Regularization (L1/L2/Elastic Net) penalizes complexity directly in the loss function
• Always compare train vs. test performance before trusting a model's score