Support Vector Machines, explained from first principles

What is a Support Vector Machine?

A Support Vector Machine (SVM) is a supervised learning algorithm used for classification and regression. Its job is simple to state and hard to do well: find the boundary that separates two classes of data with the widest possible gap.

Most classification algorithms are happy to find any line that separates the classes correctly. SVM is stricter. Among every line that could work, it picks the one that sits as far away as possible from the nearest points of both classes. That single design decision — maximize the margin, not just satisfy the constraint — is what gives SVM its reputation for generalizing well, even on datasets with relatively few examples.

SVM was formalized by Vladimir Vapnik and Alexey Chervonenkis in the 1960s and refined into its modern form (with soft margins and kernels) in the 1990s. It remained the default choice for text classification, bioinformatics, and image recognition for nearly two decades before deep learning became dominant — and it is still the right tool for small-to-medium, high-dimensional datasets today.

The math behind SVM

A hyperplane in feature space is defined by a weight vector w and a bias term b. A point x is classified based on which side of the hyperplane it falls on:

The margin between the two classes turns out to be 2 / ‖w‖. To maximize the margin, SVM needs to minimize the norm of w — subject to every training point being correctly classified with some breathing room:

This is a convex quadratic optimization problem, which means it has exactly one global solution — no risk of getting stuck in a bad local minimum, unlike many neural network training runs. In practice, SVM solvers work with the dual form of this problem, which replaces raw feature vectors with dot products — the detail that makes the kernel trick (section 6) possible.

Soft margin and the C parameter

Real datasets are rarely perfectly separable — a few outliers or overlapping points are normal. A hard margin SVM (zero tolerance for mistakes) would fail completely on such data, or overfit badly trying to accommodate every outlier. The soft margin SVM fixes this by allowing some points to violate the margin, at a cost controlled by the regularization parameter C.

SVM in Python with scikit-learn

Here's a complete, minimal example that trains an RBF-kernel SVM on a toy dataset, tunes C and gamma with cross-validation, and reports accuracy.

# 1. Imports
from sklearn import datasets
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report

# 2. Load data and split
X, y = datasets.load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# 3. SVM is distance-based -- always scale your features first
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)

# 4. Search for the best C and gamma with an RBF kernel
param_grid = {
    'C': [0.1, 1, 10, 100],
    'gamma': ['scale', 0.01, 0.001],
    'kernel': ['rbf']
}
grid = GridSearchCV(SVC(), param_grid, cv=5, n_jobs=-1)
grid.fit(X_train, y_train)

# 5. Evaluate on the held-out test set
best_model = grid.best_estimator_
predictions = best_model.predict(X_test)

print("Best params:", grid.best_params_)
print(classification_report(y_test, predictions))

Advantages and disadvantages

Frequently asked questions