You train a classifier to detect a rare disease, and it reports 99% accuracy. Ship it? Absolutely not — and understanding why is one of the most important instincts an ML engineer can have. If the disease appears in 1% of patients, a "model" that ignores its input and always predicts healthy is also 99% accurate, while catching exactly zero sick patients. Accuracy, the metric everyone reaches for first, is often the one that lies to you the most. The fix is to measure the two things that actually matter separately.

The confusion matrix: four outcomes

Every binary classifier's predictions fall into four buckets, and every useful metric is built from them:

  • True Positive (TP): predicted positive, actually positive — a correct catch.
  • True Negative (TN): predicted negative, actually negative — a correct pass.
  • False Positive (FP): predicted positive, actually negative — a false alarm (a Type I error).
  • False Negative (FN): predicted negative, actually positive — a miss (a Type II error).

Accuracy is just the fraction of everything you got right: Accuracy = (TP + TN) / (TP + TN + FP + FN). The problem is that this lumps the two kinds of error together and weights every case equally — which falls apart the moment your classes aren't balanced.

Why accuracy lies on imbalanced data

Google's ML Crash Course puts the failure plainly: "For heavily imbalanced datasets, where one class appears very rarely, say 1% of the time, a model that predicts negative 100% of the time would score 99% on accuracy, despite being useless."

That single sentence should change how you evaluate models. When positives are rare — fraud, disease, defects, security intrusions — the negatives dominate the accuracy score so completely that the metric barely reflects whether you catch any positives at all. And rare-but-important is exactly the situation most high-value classifiers operate in. Accuracy tells you how often you're right overall; it says nothing about whether you're right on the cases you actually care about.

Precision vs recall: two different questions

The answer is to split correctness into two metrics that ask different questions. Using spam detection as the example:

  • Precision = TP / (TP + FP) — of everything the model flagged as positive, what fraction really was? It "measures the fraction of emails classified as spam that were actually spam." Precision is about the cost of false alarms.
  • Recall (or TPR) = TP / (TP + FN) — of all the actual positives, what fraction did the model catch? It's "the fraction of spam emails that were correctly classified as spam." Recall is about the cost of misses.

The crucial insight is that these usually trade off against each other. As you lower the classification threshold to catch more positives, you also let in more false positives — so, per the same source, "precision and recall often show an inverse relationship, where improving one worsens the other." You cannot simply maximize both; you choose which error is more expensive for your problem.

  • Favor recall when a miss is catastrophic: cancer screening, fraud detection, security. You'll tolerate false alarms to avoid letting a real positive through.
  • Favor precision when a false alarm is expensive: a spam filter that dumps real mail, or a "you might also like" recommendation. Here a wrong positive erodes trust, so you'd rather miss some than cry wolf.

F1, and choosing your metric

When you need a single number that respects both, use the F1 score — the harmonic mean of precision and recall: F1 = 2 (precision recall) / (precision + recall). The harmonic mean matters here: unlike a plain average, it collapses toward the smaller value, so F1 is only high when both precision and recall are high. A model with 99% precision and 2% recall has an F1 around 4%, not 50% — exactly the penalty you want for a lopsided classifier.

python
# Illustrative (scikit-learn)
from sklearn.metrics import precision_score, recall_score, f1_score

precision_score(y_true, y_pred)  # TP / (TP + FP)
recall_score(y_true, y_pred)     # TP / (TP + FN)
f1_score(y_true, y_pred)         # harmonic mean of the two

F1 is a sensible default for imbalanced problems, but it isn't sacred: it weights precision and recall equally, which your problem may not. If a miss costs far more than a false alarm, weight recall higher (the F-beta score generalizes this). The real takeaway is to pick your metric from the cost of each error in your domain — never let a headline accuracy number make the decision for you.

Key Takeaways

  • Accuracy = (TP + TN) / total — and it's misleading on imbalanced data, where a do-nothing model can hit 99%.
  • Precision = TP / (TP + FP) answers "when I predict positive, am I right?" — the cost of false alarms.
  • Recall = TP / (TP + FN) answers "of the real positives, how many did I catch?" — the cost of misses.
  • Precision and recall trade off via the decision threshold; choose which error hurts more in your domain.
  • F1 is their harmonic mean, high only when both are high — a good default for imbalanced problems, but weight it to your actual error costs.

References

Machine LearningMetricsClassificationPrecisionRecall