#Lecture 008 — Model Evaluation

Every model in Lectures 005–007 printed a .score(). This lecture asks: is a single train/test split actually trustworthy, and is "accuracy" even the right thing to measure?

Contents

Prerequisites · Connection · Objectives · Lesson · Code Examples · Common Mistakes · Best Practices · Mini Challenge · Practice Questions · Answers · Takeaways

#Prerequisites

• Lectures 005–007 — at least one trained classifier to evaluate

#Connection With Previous Lectures

Every classifier trained so far used a single train_test_split. This lecture questions that setup (cross-validation) and the single "accuracy" metric each example printed (confusion matrix / precision / recall / F1) — the metrics this whole course has been quietly deferring.

#Learning Objectives

By the end of this lecture you will be able to:

• Explain the limitation of a single train/test split and how k-fold cross-validation addresses it
• Read a confusion matrix and identify TP, TN, FP, FN
• Compute precision, recall, and F1-score from a confusion matrix
• Explain why accuracy alone can be misleading on imbalanced data

#Lesson

#Train/Test Split & Cross-Validation

A train/test split (holdout method) estimates how a model will perform on unseen data by holding out a portion (commonly 20–30%) for testing and never training on it. It's simple and fast, but a single split can give a biased or high-variance performance estimate, especially on smaller datasets. K-fold cross-validation addresses this: split data into k equal folds, train on k−1 folds and test on the remaining one, repeat k times (each fold serving as the test set once), then average the scores. k=10 is a common default, but the right value depends on dataset size.

from sklearn.model_selection import cross_val_score, KFold
from sklearn.svm import SVC
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
kf = KFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(SVC(kernel="linear"), X, y, cv=kf)
print("Mean Accuracy:", scores.mean() * 100)

#Confusion Matrix, Precision, Recall, F1

A confusion matrix compares predicted vs. actual classes. For binary classification it has four cells: True Positive (TP), True Negative (TN), False Positive (FP) (a "Type I error"), and False Negative (FN) (a "Type II error"). From these:

• Accuracy = (TP+TN)/(TP+TN+FP+FN) — overall correctness, but misleading when classes are imbalanced.
• Precision = TP/(TP+FP) — of everything predicted positive, how much was actually positive. Matters when false positives are costly (e.g., spam filtering, fraud flags).
• Recall = TP/(TP+FN) — of everything actually positive, how much was caught. Matters when missing a positive is costly (e.g., disease screening).
• F1-score = 2·Precision·Recall/(Precision+Recall) — the harmonic mean, balancing the two.

#Code Examples

#Example 1 — Confusion Matrix and Classification Report

import numpy as np
from sklearn.metrics import confusion_matrix, classification_report

actual    = np.array(['Dog','Dog','Dog','Not Dog','Dog','Not Dog','Dog','Dog','Not Dog','Not Dog'])
predicted = np.array(['Dog','Not Dog','Dog','Not Dog','Dog','Dog','Dog','Dog','Not Dog','Not Dog'])

print(confusion_matrix(actual, predicted))
print(classification_report(actual, predicted))

classification_report prints precision, recall, and F1 per class — the standard first check on any classifier before trusting a single accuracy number.

#Common Beginner Mistakes

1. Chasing "accuracy" as the only metric — on an imbalanced dataset (e.g., 95% negative class), a model that always predicts "negative" gets 95% accuracy while being useless; precision/recall/F1 reveal this, accuracy alone hides it.

2. Using too few cross-validation folds on a small dataset — this can introduce bias; too many folds increase variance and computation cost. k=5 or k=10 are reasonable starting points.

3. Only looking at overall accuracy on a multi-class problem — classification_report's per-class breakdown often reveals that one class is being predicted far worse than the others.

#Best Practices

• Use k-fold cross-validation, not a single split, when the dataset is small enough that a single split's randomness could swing the result.
• Report precision, recall, and F1 alongside accuracy, especially for imbalanced classes.
• Pick the metric that matches the real cost of mistakes (e.g., prioritize recall for disease screening, precision for spam filtering).

#Mini Challenge

Take any classifier from Lecture 006 and compare its single train/test .score() against its mean cross_val_score across 5 folds — how much do they differ?

#Practice Questions

1. Why can a single train/test split give a misleading performance estimate?
2. What do TP, TN, FP, and FN stand for?
3. When would you prioritize recall over precision, and vice versa?
4. Why is accuracy alone a poor metric on an imbalanced dataset?

#Practice Answers

1. A single split's result depends heavily on which examples happened to land in the test set, especially on smaller datasets — it can be biased or high-variance.
2. True Positive, True Negative, False Positive, False Negative.
3. Prioritize recall when missing a positive is costly (e.g., disease screening); prioritize precision when a false positive is costly (e.g., spam filtering, fraud flags).
4. A model can get a high accuracy just by always predicting the majority class, without being useful at identifying the minority class at all.

#Key Takeaways

• K-fold cross-validation averages performance across k splits, reducing the bias/variance of a single train/test split
• Confusion matrix cells (TP/TN/FP/FN) are the basis for precision, recall, and F1
• Accuracy alone can hide poor performance on imbalanced classes
• Choose the metric that matches the real-world cost of false positives vs. false negatives