98% Accuracy Churn Model: Why It Misses Every Churner

8 min read · Last updated: October 25, 2026

A churn model with 98% accuracy can be useless. If only 2% of customers churn, a model that predicts "no churn" for everyone scores 98% accuracy and finds 0% of churners. Judge the model by recall and precision at the number of customers your team can actually contact, using only features known at scoring time.

Came here from Instagram, LinkedIn, X or YouTube? Here is the full source list you came for. Everything below is built from these, nothing else.

The interview rehearsal

• Your Model Has 98% Accuracy. It Misses Every Customer Who Leaves, Medium, October 24, 2026
• PracHub: Detect leakage and evaluate a prediction model, the interview question it practises

The docs

• scikit-learn: Metrics and scoring, model evaluation (accessed October 25, 2026)
• scikit-learn: Common pitfalls, data leakage (accessed October 25, 2026)
• Google Machine Learning Crash Course: Accuracy, precision and recall (accessed October 25, 2026)

Imagine a security guard at a building where 2 visitors in every 100 are thieves. His method is simple: he waves everyone through. By the end of the day he has been right 98 times out of 100. His record looks excellent, and he has caught nobody.

That guard is the "no churn" model. This article uses that one picture to explain what accuracy hides, how to check a churn model properly, and how to answer an ML interview question without bluffing.

Key takeaways

• With 20 churners in 1,000 customers, predicting "no churn" for everyone gives 98% accuracy and 0% recall.
• A prediction needs a contract first: who, when, what outcome, what horizon, what action.
• A feature is only safe if it was available at scoring time, not just timestamped before it.
• Compare models at your review budget (for example 30 cases), not at a default threshold.
• Finding churners is not saving them. Only a controlled experiment measures that.
• With 20 positive examples, you can learn the mechanics but not claim a lasting improvement.

Why does a 98% accuracy churn model find no churners?

Accuracy counts correct predictions across all customers, and most customers stay. When 98% of rows are "stays", a model that always says "stays" is right 98% of the time. It never flags a churner, so recall, the share of real churners found, is 0%.

Google's Machine Learning Crash Course makes the same point: a model that predicts negative 100% of the time "would score 99% on accuracy, despite being useless" when the positive class is rare. The Medium rehearsal uses 1,000 customers, 20 of whom churn and 980 of whom stay.

Here is what each metric says about that do-nothing baseline.

Metric	Question it answers	Formula	"Predict no churn" baseline
Accuracy	How many predictions were right?	(TP + TN) / all	98%
Recall	How many real churners did we find?	TP / (TP + FN)	0%
Precision	Of those we flagged, how many really churned?	TP / (TP + FP)	Undefined, nobody flagged

Precision is undefined here because the model flags nobody, so the denominator is zero. The definitions are from the Google crash course; the numbers are from the Medium rehearsal.

How do you evaluate a churn model, step by step?

Start by writing down the decision the model supports, then test the evidence against it. In order: define the prediction, set a baseline, audit features and labels, pick an operating rule under your review budget, and name what is still uncertain. The Medium rehearsal calls this a five-part decision note.

1. Define the prediction. Write the population, the scoring time, the label, the horizon and the action. In the rehearsal: existing customers eligible for outreach, scored every Monday using information available by 09:00, churn meaning no order in the following 28 days, and a team that can review 30 cases per cohort.
2. Set a reference. Report class counts and a simple baseline on the same future cohort. Here that is the 98% baseline with 0% recall.
3. Audit the evidence. Check that every feature existed at prediction time, that training labels had finished arriving, and that preprocessing learned only from training data.
4. Choose the operating rule. Pick a review budget or threshold, then report precision, recall, workload and cost.
5. Name the uncertainty. Say what you have not shown: later cohorts, new customers, and whether outreach changes behaviour.

How does future information leak into a churn model?

Data leakage is when information that would not exist at prediction time ends up in the model. scikit-learn, the Python machine learning library, defines it exactly that way and warns that it produces "overly optimistic performance estimates" that then fail on novel data in production.

The rehearsal gives three leaks, from blatant to subtle:

• The label in disguise. A feature like orders_next_28_days is the answer. The safe version counts orders in the preceding 28 days.
• The late arrival. A Sunday event that reaches the feature system at 10:00 Monday is too late for a 09:00 prediction, even though its event time looks fine. Audit both the event-time cutoff and the availability-time cutoff.
• The rewritten history. Joining an old prediction to the current account status can import a cancellation that happened later.

Preprocessing leaks too. scikit-learn's guide says transformations "are only learnt from the training data" and recommends splitting first, then using a Pipeline so each step is fitted on the right subset. This applies to imputation, scaling and feature selection alike.

What is label timing, and why does it break a split?

A training row is only usable if its outcome window had finished by the simulated cutoff. A row scored one week before validation starts has a 28-day window that crosses into the validation period, so its label is not yet known and must not be used for training.

The rehearsal adds two practical rules. Use only examples whose outcome windows have completed and whose labels have arrived, including reporting delays. And build eligibility from timestamps, because a fixed row-count gap is not automatically a 28-day gap. Keep a final future period untouched for the last evaluation. The goal, in the article's words, is to "reproduce the situation in which the model will be used."

Why can the better churn model have lower accuracy?

Because accuracy rewards the many customers who stay, while the business cares about the few who leave. A model that flags more customers finds more churners but raises more false alarms, so its accuracy drops. Compare models by precision, recall and workload at the same review budget.

The rehearsal's illustrative numbers show the trade-off with 20 churners in 1,000 customers:

Setting	Flagged	Churners found	False alarms	Missed	Precision	Recall	Accuracy
A	30	12	18	8	40%	60%	97.4%
B	80	16	64	4	20%	80%	93.2%

Setting B finds four more churners and has the lower accuracy. But your team can only review 30 cases. B flags 80, so it does not fit the budget, and its top 30 would need a separate evaluation. Note the scale too: with 20 churners, one extra discovery moves recall by five percentage points.

When should you use this approach, and when not?

Use precision, recall and a review budget whenever the positive class is rare and an action follows each flag. Skip the heavy machinery when classes are balanced or the decision is not a ranked list. Accuracy is a fine metric when both outcomes are common and mistakes cost about the same.

Situation	What to do
Rare positives (1% to 5%) and a team that acts on flags	Report recall and precision at the review budget
Roughly balanced classes	Accuracy is acceptable. scikit-learn notes balanced accuracy equals accuracy on balanced data
One mistake costs far more than the other	Optimise for the metric that tracks that mistake, as Google's crash course advises
Probabilities drive decisions, such as discount size	Add a calibration check on predicted-risk groups
You must claim customers were saved	Run a controlled experiment, not an offline score

What does a small example look like in code?

The snippet below reproduces the 98% baseline, then scores any model at a fixed review budget. It uses scikit-learn's DummyClassifier, a baseline model, plus a helper that takes the top k customers by risk score.

import numpy as np
import pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, recall_score, precision_score

# 1,000 customers, 20 churners (1 = churned)
y = np.array([1] * 20 + [0] * 980)
X = np.zeros((len(y), 1))

baseline = DummyClassifier(strategy="most_frequent").fit(X, y)
pred = baseline.predict(X)

print(accuracy_score(y, pred))                       # 0.98
print(recall_score(y, pred))                         # 0.0
print(precision_score(y, pred, zero_division=0))     # 0.0 (really undefined)

def at_budget(y_true, scores, k=30):
    """Precision and recall if the team reviews only the top k scores."""
    top = np.argsort(scores)[::-1][:k]
    found = y_true[top].sum()
    return {"flagged": k, "found": int(found),
            "precision": found / k, "recall": found / y_true.sum()}

# Label timing: keep only rows whose 28-day outcome window has finished
horizon = pd.Timedelta(days=28)
train = df[df["scored_at"] + horizon <= validation_start]

If at_budget finds 12 churners among the top 30, you get 40% precision and 60% recall, which matches Setting A above. For preprocessing, wrap the steps and the model in a scikit-learn Pipeline so each fit only sees training data.

What can offline evidence not tell you?

It cannot tell you that outreach will save anyone. Finding 12 churners does not show that contacting them keeps 12 customers, because some would have stayed anyway. Measuring the effect of the intervention needs a controlled experiment, with some flagged customers contacted and some not.

This is the honest part of the interview answer, and it is where many candidates stop short. The rehearsal lists the limits plainly:

• Twenty positives is tiny. It can teach the mechanics but cannot support a confident claim of persistent improvement.
• New customers need their own holdout. Claims about unseen customers need a customer-level holdout, with cold-start users reported separately.
• Probabilities need checking. If scores drive decisions, test calibration across risk groups.

Saying "here is what I have not shown" is not a weak answer. It tells the interviewer you know the difference between a metric and a decision. The same instinct shows up in good system design interviews: state the constraints, then the trade-offs.

How do you rehearse this in 30 minutes?

The Medium rehearsal splits half an hour into four blocks. Practise out loud, with a timer, on any churn-style question.

1. Minutes 0 to 5: define the prediction contract.
2. Minutes 5 to 15: audit features and label timing.
3. Minutes 15 to 25: choose a decision rule under a limited review budget.
4. Minutes 25 to 30: explain what offline evidence cannot establish.

The rehearsal ties to a PracHub question about detecting leakage and evaluating a prediction model, which is a good place to find a prompt to practise on.

FAQ

Is 98% accuracy good for a churn model?

Not by itself. If 98% of customers stay, always predicting "stays" scores 98% too. Ask for the class counts, the baseline on the same cohort, and recall and precision at the number of customers your team can contact.

What is the difference between precision and recall?

Precision is the share of flagged customers who really churn: TP / (TP + FP). Recall is the share of all real churners you caught: TP / (TP + FN). Per scikit-learn, precision avoids false alarms and recall finds all the positives.

What is data leakage in machine learning?

Data leakage is using information at training or evaluation time that would not be available when the model makes real predictions. It inflates scores, for example in cross-validation, and the model then performs worse in production. In churn work it often hides in features built from future orders or later account status.

Does a high churn score mean we can save those customers?

No. A model that finds churners shows prediction, not retention. To know whether outreach works, run a controlled experiment that compares contacted and uncontacted flagged customers.

Metrics are only half of AI work, and the other half is judgement. Keep sharpening both with the free tools on Art of Code, and read code was never the hard part for why deciding beats typing.

Written by Adithya Guttha, Founder of Art of Code.