#Lecture 006 — Classification Algorithms

Four more classification algorithms, each with a different way of drawing the line between classes: decision trees (ask questions), KNN (look at neighbors), SVM (maximize the margin), and Naive Bayes (apply probability).

Contents

Prerequisites · Connection · Objectives · Lesson · Code Examples · Common Mistakes · Best Practices · Mini Challenge · Practice Questions · Answers · Takeaways

#Prerequisites

• Lecture 005 — logistic regression, the first classifier in the course
• Lecture 004 — scaled features (matters for KNN and SVM here)

#Connection With Previous Lectures

Logistic regression (Lecture 005) was one way to classify. This lecture covers four more, each with different tradeoffs in interpretability, speed, and how they handle non-linear boundaries — all of which get formally compared in Lecture 008's evaluation metrics.

#Learning Objectives

By the end of this lecture you will be able to:

• Explain how a decision tree splits data, and name its two common purity measures
• Explain KNN's "lazy learning" approach and why K must be chosen carefully
• Explain what SVM maximizes and what the "kernel trick" does
• Explain Naive Bayes' core assumption and why it still works well despite being unrealistic

#Lesson

#Decision Trees

A decision tree makes predictions by asking a sequence of yes/no (or threshold) questions about features, structured as a flowchart: root node → internal nodes (tests) → leaf nodes (final prediction). Splits are chosen to make each resulting group "purer." Two common purity measures: Entropy (0 = pure, higher = more mixed; drives Information Gain = how much a split reduces entropy) and the Gini Index (1 − Σpᵢ²; lower is purer; scikit-learn's default criterion). Trees are easy to interpret and need little preprocessing, but un-pruned trees are prone to overfitting.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

clf = DecisionTreeClassifier(criterion="gini", max_depth=4, random_state=42)
clf.fit(X_train, y_train)
print("Accuracy:", clf.score(X_test, y_test))

#K-Nearest Neighbors (KNN)

KNN is a "lazy learner" — it does no real training, just stores the data, and at prediction time finds the K closest training points (by a distance metric) to the new point. For classification, it takes a majority vote among those K neighbors. Common distance metrics: Euclidean (straight-line), Manhattan (grid-like), Minkowski (general form; p=2 gives Euclidean, p=1 gives Manhattan). Choosing K matters: too small overfits to noise, too large underfits/oversmooths; an odd K avoids ties in binary classification.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

knn = KNeighborsClassifier(n_neighbors=5, metric="euclidean")
knn.fit(X_train, y_train)
print("Accuracy:", knn.score(X_test, y_test))

Beginner pitfall: forgetting to scale features before KNN (or SVM) — since both are distance-based, a feature on a 0–10,000 scale will dominate one on a 0–1 scale.

#Support Vector Machines (SVM)

SVM finds the decision boundary (hyperplane) that best separates classes by maximizing the margin — the distance between the hyperplane and the closest points of each class, called support vectors. A "hard margin" demands perfect separation (sensitive to outliers); a "soft margin" (controlled by parameter C) tolerates some misclassification — higher C penalizes mistakes more strictly. When data isn't linearly separable, a kernel (linear, polynomial, or RBF) implicitly projects it into higher dimensions where a linear split works — the "kernel trick."

from sklearn.datasets import load_breast_cancer
from sklearn.svm import SVC

cancer = load_breast_cancer()
X = cancer.data[:, :2]
y = cancer.target

svm = SVC(kernel="linear", C=1)
svm.fit(X, y)

#Naive Bayes

Naive Bayes classifies by applying Bayes' theorem: the probability of a class given the features is proportional to the probability of the features given the class, times the prior probability of the class. The "naive" part is the assumption that all features are independent of each other given the class — but despite this simplification, it works surprisingly well in practice, especially for text. Variants: Gaussian (continuous features), Multinomial (count data, e.g., word frequencies), Bernoulli (binary features, e.g., word present/absent).

#Code Examples

The three runnable examples above (decision tree, KNN, SVM) all use the same fit()/predict()/score() shape from Lecture 003 — the main differences are the import and the constructor's hyperparameters (max_depth, n_neighbors, kernel/C).

#Common Beginner Mistakes

1. Forgetting to scale features before KNN or SVM — both are distance-based and get distorted by unscaled features (carried over from Lecture 004).

2. Treating "SVM" as one algorithm rather than a family — it behaves very differently depending on the kernel/C choice; an untuned RBF-kernel SVM can wildly overfit or underfit.

3. Expecting Naive Bayes to fail because its independence assumption is "obviously wrong" for real data — in practice it's still a strong, fast baseline, especially for spam filtering and text classification, because classification only needs the ranking of class probabilities to be right, not their exact values.

#Best Practices

• Start with a decision tree or logistic regression as an interpretable baseline before trying SVM or ensembles.
• Always scale features before KNN/SVM; tree-based models don't need it.
• Use cross-validation (Lecture 008) to pick K, C, and kernel rather than guessing once.

#Mini Challenge

Run all three code examples (tree, KNN, SVM) on the same load_iris split and compare their accuracy scores — note which one changes the most if you remove the random_state and rerun.

#Practice Questions

1. What two purity measures does a decision tree commonly use to choose splits?
2. Why is KNN called a "lazy learner"?
3. What does SVM maximize when choosing a decision boundary?
4. What is the "naive" assumption in Naive Bayes, and why doesn't it ruin the algorithm in practice?

#Practice Answers

1. Entropy (via Information Gain) and the Gini Index.
2. It does no real training — it just stores the data and does all the work (finding nearest neighbors) at prediction time.
3. The margin — the distance between the hyperplane and the closest points of each class (the support vectors).
4. That all features are independent of each other given the class; it works well anyway because classification only needs probabilities ranked correctly, not exactly right.

#Key Takeaways

• Decision trees split on purity (entropy/Gini); easy to interpret, prone to overfitting unpruned
• KNN classifies by majority vote among the K nearest points; needs scaled features and a carefully chosen K
• SVM maximizes the margin between classes; kernels let it handle non-linear boundaries
• Naive Bayes applies Bayes' theorem with an independence assumption that's unrealistic but works well in practice