#Lecture 005 — Regression: Linear & Logistic

Our first two trainable models. Linear regression predicts a number; logistic regression — despite the name — predicts a category. Both are fit using the gradient-descent intuition from Lecture 002.

Contents

Prerequisites · Connection · Objectives · Lesson · Code Examples · Common Mistakes · Best Practices · Mini Challenge · Practice Questions · Answers · Takeaways

#Prerequisites

• Lecture 004 — clean, preprocessed X/y data to train on
• Lecture 002 — gradient descent intuition

#Connection With Previous Lectures

This is the first lecture that actually trains a model on preprocessed data. Logistic regression's probability output also sets up Lecture 008's confusion matrix / precision / recall, which are scored on predicted class probabilities.

#Learning Objectives

By the end of this lecture you will be able to:

• Fit and read the output of a linear regression model
• Explain why logistic regression is a classification algorithm despite its name
• Describe the sigmoid function's role in logistic regression
• State one key difference between how each model is fit

#Lesson

#Linear Regression

Linear regression predicts a continuous number by fitting a straight line (or hyperplane, with multiple features) through the data. For one feature: ŷ = θ₀ + θ₁x (intercept + slope × input). With multiple features: ŷ = θ₀ + θ₁x₁ + θ₂x₂ + ... The "best" line is the one that minimizes the Mean Squared Error between predictions and actual values, typically found via gradient descent or the closed-form least-squares solution.

Key assumptions taught: linearity, independent errors, constant error variance (homoscedasticity), normally distributed errors, no severe multicollinearity between features, no autocorrelation, and additive effects.

import numpy as np
from sklearn.linear_model import LinearRegression

np.random.seed(42)
X = np.random.rand(50, 1) * 100
Y = 3.5 * X + np.random.randn(50, 1) * 20   # true slope 3.5, plus noise

model = LinearRegression()
model.fit(X, Y)
print("Slope:", model.coef_[0][0], "Intercept:", model.intercept_[0])

Beginner pitfall: assuming linear regression can model any trend. If the true relationship is curved, a straight-line fit underfits — a preview of Lecture 009.

#Logistic Regression

Despite the name, logistic regression is a classification algorithm. It computes a weighted sum of the inputs (z = w·x + b) and squashes it through the sigmoid function, σ(z) = 1/(1+e⁻ᶻ), producing a value between 0 and 1 interpreted as a probability. A threshold (commonly 0.5) converts that probability into a class label. Binomial logistic regression handles two classes; multinomial (using softmax) handles 3+ unordered classes; ordinal handles 3+ ordered classes.

#Code Examples

#Example 1 — Logistic Regression on a Real Dataset

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=23)

clf = LogisticRegression(max_iter=10000, random_state=0)
clf.fit(X_train, y_train)
print(f"Accuracy: {accuracy_score(y_test, clf.predict(X_test)) * 100:.2f}%")

max_iter is raised here because logistic regression is fit iteratively (gradient-based); the default iteration count can be too low to converge on some datasets, which scikit-learn warns about if it happens.

#Common Beginner Mistakes

1. Thinking "regression" in the name means logistic regression predicts continuous values like linear regression does — it predicts class probabilities and is firmly a classification tool.

2. Not checking linear regression's assumptions — fitting a straight line to a clearly curved relationship produces a model that looks "trained" but is systematically wrong.

3. Misreading the sigmoid threshold as fixed at 0.5 — 0.5 is just the common default; it's tunable, and should be moved when false positives and false negatives have very different costs (previewed properly in Lecture 008).

#Best Practices

• Plot the data before fitting linear regression — a quick scatter plot reveals whether a straight line is even a reasonable model.
• For logistic regression, look at predicted probabilities, not just the final class label, to judge how confident the model actually is.

#Mini Challenge

Run Example 1 and print clf.predict_proba(X_test)[:5] alongside clf.predict(X_test)[:5] — compare how the probabilities relate to the final 0/1 predictions.

#Practice Questions

1. Why is logistic regression classified as a classification algorithm?
2. What does the sigmoid function output, and what range is it in?
3. Name two assumptions linear regression makes about the data.
4. What fitting method is used for logistic regression, vs. least squares for linear regression?

#Practice Answers

1. It predicts class probabilities (via the sigmoid function) and a class label, not a continuous numeric quantity.
2. A value between 0 and 1, interpreted as a probability.
3. Any two of: linearity, independent errors, constant error variance, normally distributed errors, no severe multicollinearity, no autocorrelation.
4. Maximum likelihood, vs. least squares for linear regression.

#Key Takeaways

• Linear regression predicts a number by fitting a line that minimizes squared error
• Logistic regression predicts a class via the sigmoid function, despite the "regression" name
• Both are trained with gradient-descent-style optimization
• A straight-line fit on a curved relationship underfits — foreshadows Lecture 009