Precision, recall & a baseline

Understand the concept

Accuracy is the fraction of correct predictions. For rare positive cases, a model predicting only the majority class can achieve high accuracy while detecting no positives. Compare against a simple baseline before treating a model as useful.

With class 1 as positive, precision = TP / (TP + FP): how many predicted positives were correct? Recall = TP / (TP + FN): how many actual positives were found? F1 is their harmonic mean. A confusion matrix exposes the counts behind these metrics.

Choose metrics with the use case, label balance, and error costs in mind. Changing a threshold can change precision and recall; select it using validation data. For undefined ratios, document the convention. This example defines a zero denominator as a score of 0 and uses only the standard library.

actual = [1, 1, 1, 0, 0, 0]
predicted = [1, 0, 1, 1, 0, 0]
tp = sum(a == 1 and p == 1 for a, p in zip(actual, predicted))
fp = sum(a == 0 and p == 1 for a, p in zip(actual, predicted))
fn = sum(a == 1 and p == 0 for a, p in zip(actual, predicted))
precision = tp / (tp + fp) if tp + fp else 0
recall = tp / (tp + fn) if tp + fn else 0
print(tp, fp, fn)
print(f"Precision: {precision:.2f}, Recall: {recall:.2f}")

Output / what to expect

2 1 1
Precision: 0.67, Recall: 0.67

Try it yourself

For 100 examples with 5 positives, an all-negative model gets how much accuracy and positive-class recall? What does accuracy hide?

accuracy = 95 / 100
recall = 0 / 5
print(accuracy, recall)  # 0.95 0.0
# High accuracy hides the fact that every positive was missed.

Concept samjho

Accuracy correct predictions ka fraction hai. Agar positive class rare hai toh sirf majority class predict karne wala model high accuracy paa sakta hai, lekin ek bhi positive detect nahi karega. Model ko useful kehne se pehle simple baseline se compare karo.

Class 1 ko positive maan kar precision = TP / (TP + FP): predicted positives mein kitne sahi the? Recall = TP / (TP + FN): actual positives mein kitne mile? F1 in dono ka harmonic mean hai. Confusion matrix metrics ke peeche ki counts dikhata hai.

Use case, class balance aur galti ki cost ke hisaab se metrics choose karo. Threshold badalne se precision aur recall badal sakte hain; selection validation data par karo. Undefined ratio ka convention clear batao. Example mein denominator zero ho toh score 0 hai, aur standard library kaafi hai.

actual = [1, 1, 1, 0, 0, 0]
predicted = [1, 0, 1, 1, 0, 0]
tp = sum(a == 1 and p == 1 for a, p in zip(actual, predicted))
fp = sum(a == 0 and p == 1 for a, p in zip(actual, predicted))
fn = sum(a == 1 and p == 0 for a, p in zip(actual, predicted))
precision = tp / (tp + fp) if tp + fp else 0
recall = tp / (tp + fn) if tp + fn else 0
print(tp, fp, fn)
print(f"Precision: {precision:.2f}, Recall: {recall:.2f}")

Output / kya expect karein

2 1 1
Precision: 0.67, Recall: 0.67

Ab khud try karo

100 examples mein 5 positives hain. All-negative model ki accuracy aur positive-class recall kya hogi? Accuracy kya chhupa rahi hai?

accuracy = 95 / 100
recall = 0 / 5
print(accuracy, recall)  # 0.95 0.0
# High accuracy hides the fact that every positive was missed.