Evaluation Metrics for Machine Learning Models

The Lying Spam Filter

Imagine a spam filter that has learned one perfect rule: mark every single email as spam. On a dataset where 95% of emails are spam, this filter achieves 95% accuracy. That sounds excellent. But your inbox is completely empty. Every legitimate email — your job offers, bank statements, messages from family — is silently discarded.

Accuracy told you 95%. Reality told you the filter is useless.

This is why choosing the right evaluation metric is not a technicality — it is the difference between a model that works and one that silently fails.

The Confusion Matrix: Ground Truth

Every classification model's performance starts here. For a binary problem (Positive / Negative), every prediction falls into one of four cells:

                 Predicted Positive   Predicted Negative
Actual Positive |   True Positive   |   False Negative  |
Actual Negative |   False Positive  |   True Negative   |

• True Positive (TP): Model said positive. It was positive. Correct.
• True Negative (TN): Model said negative. It was negative. Correct.
• False Positive (FP): Model said positive. It was negative. Wrong. (Type I error)
• False Negative (FN): Model said negative. It was positive. Wrong. (Type II error)

Every metric you will ever use is derived from these four numbers.

The Core Metrics — Formulas and Meaning

Accuracy answers: "Of all predictions, what fraction were correct?"

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Useful when classes are balanced. Misleading on imbalanced data (the spam filter problem).

Precision answers: "Of everything I labeled positive, how many actually were?"

Precision = TP / (TP + FP)

Prioritize precision when false positives are costly. Example: a cancer screening that triggers expensive, invasive follow-up surgery — you do not want false alarms.

Recall (Sensitivity) answers: "Of all actual positives, how many did I catch?"

Recall = TP / (TP + FN)

Prioritize recall when false negatives are costly. Example: a disease detection system where missing a sick patient has severe consequences.

F1 Score is the harmonic mean of precision and recall:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

F1 punishes extreme imbalance between precision and recall. A model with 100% precision and 0% recall gets F1 = 0, not 50%. Use F1 when you need a single score that balances both.

ROC Curve and AUC: Threshold-Independent Performance

Most classifiers output a probability (e.g., 0.73 = 73% chance of being positive). The decision threshold determines where you draw the line between positive and negative. At 0.5 by default.

The ROC Curve (Receiver Operating Characteristic) plots:

• True Positive Rate (Recall) on the Y-axis
• False Positive Rate (FP / (FP + TN)) on the X-axis

as the threshold varies from 0 to 1. A perfect model reaches the top-left corner (TPR=1, FPR=0). A random model follows the diagonal.

AUC (Area Under the Curve) summarizes the ROC curve in one number:

• AUC = 1.0: Perfect model
• AUC = 0.5: Random guessing (useless)
• AUC = 0.85: Good, practical model

AUC is threshold-independent — it measures how well the model ranks positives above negatives, regardless of where you set the cutoff.

Regression Metrics

For problems with continuous outputs (predicting house prices, temperature, etc.):

MAE (Mean Absolute Error): Average absolute difference between predictions and actuals. Robust to outliers. Easy to interpret (in the same units as the target).

MAE = (1/n) × Σ |y_i - ŷ_i|

MSE (Mean Squared Error): Average squared difference. Penalizes large errors heavily. Used in many loss functions.

MSE = (1/n) × Σ (y_i - ŷ_i)²

RMSE (Root Mean Squared Error): Square root of MSE. Same units as the target, more sensitive to large errors than MAE.

RMSE = √MSE

R² (R-Squared / Coefficient of Determination): Fraction of variance in the target that the model explains. R²=1 is perfect; R²=0 means the model does no better than predicting the mean; R²<0 means the model is worse than predicting the mean.

R² = 1 - (SS_residual / SS_total)

Complete sklearn Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (
    classification_report, confusion_matrix,
    roc_auc_score, roc_curve, ConfusionMatrixDisplay
)
import matplotlib.pyplot as plt

# Setup
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
y_pred  = model.predict(X_test)
y_proba = model.predict_proba(X_test)[:, 1]

# --- Confusion Matrix ---
cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:")
print(cm)
# Output:
# [[40  3]
#  [ 2 69]]

# --- Classification Report ---
print("\nClassification Report:")
print(classification_report(y_test, y_pred,
      target_names=['Malignant', 'Benign']))
# Output:
#               precision  recall  f1-score  support
#    Malignant      0.95    0.93      0.94       43
#        Benign     0.96    0.97      0.97       71
#     accuracy                        0.96      114
#    macro avg      0.96    0.95      0.95      114
# weighted avg      0.96    0.96      0.96      114

# --- ROC-AUC ---
auc = roc_auc_score(y_test, y_proba)
print(f"\nROC-AUC Score: {auc:.4f}")
# Output: ROC-AUC Score: 0.9957

fpr, tpr, _ = roc_curve(y_test, y_proba)
plt.figure(figsize=(7, 5))
plt.plot(fpr, tpr, color='steelblue', lw=2, label=f'ROC Curve (AUC = {auc:.3f})')
plt.plot([0, 1], [0, 1], 'k--', label='Random Classifier')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate (Recall)')
plt.title('ROC Curve — Breast Cancer Detection')
plt.legend()
plt.tight_layout()
plt.savefig('roc_curve.png', dpi=150)
plt.show()

Handling Imbalanced Classes

When one class dominates (e.g., fraud detection: 99% non-fraud, 1% fraud), most classifiers learn to ignore the minority class. Solutions:

class_weight='balanced': Automatically adjusts the loss function so that the minority class gets more weight during training.

model = LogisticRegression(class_weight='balanced', max_iter=1000)

Prefer F1 score or ROC-AUC over accuracy for evaluation on imbalanced data.

Metric Quick Reference Table

Key Takeaways

• Accuracy lies on imbalanced data. Always check the confusion matrix first.
• Precision vs Recall is a tradeoff driven by domain: what costs more, false alarms or missed detections?
• F1 Score is the go-to single metric for imbalanced classification.
• ROC-AUC measures ranking quality independent of threshold — ideal for model comparison.
• For regression, MAE is interpretable; RMSE punishes outliers; R² shows explained variance.
• Use class_weight='balanced' as a first-line fix for class imbalance.