Every spring we get a version of the same phone call. An inspection system that has run quietly since commissioning is suddenly rejecting good parts — or, worse, a customer has found defects that the system should have caught. Nobody changed the model. Nobody touched the camera. “It just got worse.”
It almost never just gets worse. A vision model is a frozen snapshot of the world it was trained on: one set of lighting conditions, one batch of material, one tooling state, one season. The world keeps moving. The gap between the snapshot and the present is what machine-learning engineers call drift, and on a factory floor it has very physical causes.
This article walks through where drift comes from, how to see it before your customers do, and what a sensible maintenance routine for a model looks like.
The model didn’t change. Everything around it did.
Think of a trained inspection model as a gauge that was calibrated on one particular day. It is accurate for conditions like that day. When conditions shift far enough, the readings shift with them — without any visible fault.
Light is the first suspect
The camera sees light, not parts. Anything that changes the light reaching the sensor changes the image the model receives:
- Ambient daylight. Skylights and open dock doors add light that varies by hour and by season. A cell commissioned in February may see a very different scene at 4 p.m. in July.
- Aging illumination. LED output slowly declines over its life, and phosphor-converted white LEDs can shift color as they age. A strobe that is a few percent dimmer moves every pixel value with it.
- Dirty optics. Dust, oil mist and washdown residue on a lens or a protective window soften edges and lower contrast — slowly enough that nobody notices from day to day.
Then the material
Your parts are not identical over time, even when they are within specification:
- A new resin lot with a slightly different gloss.
- A coil of steel from a second mill with a different surface finish.
- A packaging film supplier change that alters how a seal reflects light.
- Humidity changing how paperboard or food product looks and behaves.
To a person these are trivial. To a model trained on the old material, a different gloss can look like a defect — or, more dangerously, a real defect can look more like the new “normal”.
Then the machine and the process
Tooling wears, fixtures loosen and line speeds get adjusted. A part that arrives 3 mm further from the camera, or a millisecond later relative to the strobe, is a slightly different image. Maintenance that moves a camera bracket by a few degrees can be enough.
Why July?
In many plants, summer is when several of these changes happen at once: the most daylight, the highest humidity, the hottest enclosures, and often a shift in product mix or temporary staff. That is why a model can look healthy for months and then slip within a few weeks.
Early warnings and late warnings
The most expensive way to discover drift is a customer complaint. By then the defective parts have shipped. The goal is to watch signals that move before detection performance drops.
| Signal | What it tells you | How early |
|---|---|---|
| Image statistics (brightness, contrast, sharpness) | The scene the camera sees has changed | Very early |
| Model confidence distribution | Inputs are becoming less like the training data | Early |
| Reject rate per defect class | One class is drifting faster than others | Early to medium |
| False-reject audits by operators | The model is over-rejecting good parts | Medium |
| Escapes found downstream or by customers | Detection has already failed | Late |
Watch the confidence, not just the verdict
A model that says “good” with 99% confidence and a model that says “good” with 61% confidence produce the same verdict and very different risk. When the average confidence on good parts slides, or the share of borderline decisions grows week after week, the model is telling you that the parts in front of it look less like the ones it learned from.
Watch the image itself
Simple statistics on every frame — mean brightness, contrast, a focus measure — are cheap to compute on the edge device and surprisingly informative. A slow fall in contrast over six weeks is usually a dirty window or an aging light. A daily oscillation in brightness is almost always daylight.
Set a drift budget before you go live
The single most useful thing you can do is decide, in writing and before commissioning, what “too much change” means. We call this a drift budget. It has three parts:
- A baseline. The detection rate, false-reject rate and confidence distribution measured at site acceptance, on a labeled validation set kept for exactly this purpose.
- Thresholds. For example: alert if mean confidence on good parts drops more than five points from baseline for three consecutive days, or if any defect class’s reject rate doubles.
- A response. Who is notified, who reviews sample images, and what triggers a retraining job.
Without a budget, every conversation about model performance becomes an argument about anecdotes. With one, it is a measurement.
Retraining is maintenance, not failure
Plants accept that gauges need calibration and bearings need grease. Models need the same routine attention. A healthy retraining process looks like this:
- Collect the right images. Pull borderline and misclassified images from the period since the last training — these are the most valuable examples.
- Label them carefully. A quality engineer confirms each label. Mislabeled data is the fastest way to make drift worse.
- Retrain and compare. Train a candidate model and evaluate it against the original validation set and a new set from current conditions. The candidate must not regress on the old data.
- Deploy safely. Run the candidate in shadow mode next to the live model, then switch over using an A/B update that can be rolled back in seconds.
- Record it. Update the model card: what changed, which data was added, and what the measured performance is now.
Design drift out where you can
Monitoring catches drift; good engineering reduces how much of it you have to catch:
- Control the light. Enclosures and shrouds that block ambient light remove the largest single source of seasonal drift. Strobed illumination that overpowers ambient light helps too. Our article on lighting geometry covers the trade-offs.
- Train on variety. Deliberately include different lots, shifts and seasons in training data, and use synthetic variation carefully to cover conditions you haven’t seen yet.
- Pick the right model type. Anomaly-detection models that learn “normal” can flag new kinds of change rather than silently misclassifying them — see anomaly detection vs supervised classification.
- Keep the optics clean. Add window cleaning to the preventive-maintenance schedule, with a reference image to compare against.
What this looks like in practice
On systems covered by Perflens Care, each Gatepoint computes image statistics and confidence distributions on-device and reports them to Perflens Console. Drift alerts follow the budget agreed at commissioning, and retraining runs on a schedule set by the care tier — from twice a year to continuously. Nothing about this requires sending production images to a cloud service; the monitoring runs where the camera is.
The point is not the tooling. It is the habit: treat a model as an instrument that needs calibration, measure it against a baseline, and fix it before July does it for you.