What data drift is, and why your model fails without breaking

There's a failure mode that never shows up in a log. The model responds, the API returns 200, the dashboard renders. And the predictions get worse every month.

That's data drift: the data arriving at your model in production stops resembling the data it was trained on. The model isn't broken. It's confidently answering questions about a world that no longer exists.

Why you don't see it

Traditional software fails loudly. An exception, a timeout, a 500. You have alerts for that.

A machine learning model fails quietly, and that difference is the one almost nobody internalises in time. A scoring model trained on 2024 data still returns a number in 2026. The number has the right shape, arrives on time, and sits in the usual range. It just no longer describes reality.

You find out months later, and not from the system — from the business result. Defaults went up. Conversion went down. Sales says the recommendations "feel off". By the time someone connects the dots, you've spent two quarters making decisions on a model that stopped working.

The three types that matter

Not all drift is the same, and confusing them leads to monitoring the wrong thing.

• Data drift. The distribution of input features changes. 70% of your customers used to come through a branch; now 70% come through the app. Same features, different distribution.
• Concept drift. The relationship between inputs and what you're predicting changes. The feature is distributed the same way, but it no longer means the same thing. The pandemic did this to nearly every demand model on the planet.
• Label drift. The distribution of what you predict changes. If you trained on 3% fraud and reality is now 8%, your model is calibrated for a world that isn't this one.

Concept drift is the most expensive, because it's the hardest to see: your inputs look perfectly normal.

What to actually monitor

Monitoring a model is not checking whether the service is up. Your infrastructure monitoring already does that, and it tells you nothing about whether the model is right.

What to watch:

1. The distribution of every input feature, against the training distribution. Statistical tests help here — Kolmogorov-Smirnov for continuous features, chi-squared for categorical ones.
2. The distribution of predictions. If your model starts saying "no" far more than before and nothing in the business explains it, something moved.
3. Actual performance against ground truth, when you have it. It's the most honest signal and the slowest: a default-risk model can take 90 days to tell you whether you were right.
4. Input volume and quality. Nulls that weren't there before, new categories, a field that's been arriving empty since the app team changed the form.

The uncomfortable part: you can't wait for ground truth

In most business cases the truth arrives late. You know whether the loan was repaid three months later. You know the customer churned once they've churned.

That's why drift monitoring isn't a nice-to-have: it's your only early signal. It's the difference between finding out in two weeks and finding out in two quarters.

And in a regulated sector, you also have to defend it

If you work in banking or insurance, this stops being an engineering best practice and becomes a risk matter. When the committee asks why the model decided what it decided, "that's what it returned" isn't an answer. You need the model version, the data version, the metrics at that moment, and evidence that someone was watching.

None of that can be improvised on the day they ask. Either it was there from the start, or it isn't there.

What to do if you have a model in production with no monitoring

Nothing dramatic. The highest-return first step is almost never retraining — it's instrumenting.

Take the model you already have, log its inputs and outputs, compare them against the training distribution, and set a threshold that fires an alert. That work is usually weeks, not months, and it turns an invisible risk into a number somebody looks at.

Then we can argue about retraining. But first you have to be able to see it.