Model Evaluation Metrics: Confusion Matrix, Precision, Recall, F1 and Cross-Validation

A model that screens chest X-rays reports 95% accuracy. Impressive? If only 5% of patients actually have the disease, a model that says "healthy" for everyone also scores 95% — while missing every sick patient. Choosing the right metric is not a technical detail; it decides whether a model helps or harms.

The confusion matrix

For a yes/no classifier, every prediction lands in one of four boxes. Take a model that flags fraudulent transactions:

Every classification metric is built from these four numbers.

Precision and recall

• Precision = TP / (TP + FP) — of everything we flagged, how much was really fraud? Low precision means many annoyed genuine customers.
• Recall = TP / (TP + FN) — of all real fraud, how much did we catch? Low recall means fraud slips through.

They pull against each other. Lower the threshold and you catch more fraud (recall up) but block more genuine customers (precision down).

F1 score is the harmonic mean of precision and recall — high only when both are reasonably high.

Probabilities: ROC-AUC and PR-AUC

Most classifiers output a probability. Instead of judging one threshold, you can measure ranking quality across all thresholds:

• ROC-AUC — the chance that a random positive example gets a higher score than a random negative one. 0.5 is guessing, 1.0 is perfect.
• PR-AUC (average precision) — more informative when positives are rare, like fraud.

Regression metrics recap

• MAE — average absolute error, in real units ("₹38,000 off on average").
• RMSE — penalises large misses; use when big errors are especially bad.
• R² — share of variance explained; useful for comparing models on the same data.
• MAPE — average percentage error; easy to explain, but breaks when actual values are near zero.

Cross-validation: a fairer estimate

One train/test split can be lucky or unlucky. K-fold cross-validation splits the training data into k parts (often 5), trains on k−1 and validates on the remaining one, rotating until every part has been the validation set once. You get k scores — their average is more reliable, and their spread shows how stable the model is.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix, classification_report, roc_auc_score

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

model = RandomForestClassifier(n_estimators=300, random_state=42)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X_train, y_train, cv=cv, scoring="f1")
print("CV F1: %.3f ± %.3f" % (scores.mean(), scores.std()))

model.fit(X_train, y_train)                               # final check on the untouched test set
pred  = model.predict(X_test)
proba = model.predict_proba(X_test)[:, 1]
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, digits=3))
print("ROC-AUC:", round(roc_auc_score(y_test, proba), 3))

Choosing the threshold

The default 0.5 threshold is rarely the best business choice. If missing fraud costs ₹10,000 and blocking a genuine payment costs ₹200 in support time and goodwill, pick the threshold that minimises total cost on validation data. This is how ML connects to real decisions.