A pilot that ends with “the model works” and nothing else has answered the wrong question. Industrial AI pilots rarely stall because the technology fails. They stall because they were designed to prove that a model could work, not that it pays, that the plant can support it, and that it can be copied to the next line. The fix is to treat the pilot as the first deployment: production hardware, a measured baseline, written success criteria, a security review in week one and a named owner for the model after go-live.
Before Perflens I ran maintenance on continuous-process lines and sat through my share of vendor pilots. The ones that went nowhere were rarely technical failures. Most of them ended in a polite slide deck, a few enthusiastic operators and no purchase order. This article is about why.
What a pilot is supposed to answer
There are two different questions, and mixing them is where trouble starts:
- Feasibility: can a camera and a model see this defect, or can a sensor detect this failure mode? That is a bench question. It is answered with samples, a lighting study and a baseline model, in weeks, without touching the line.
- Value: if we install this everywhere it applies, will it pay back, and can we run it for years? That is a production question. It needs the real line, the real network, the real operators and a measurement that finance will accept.
A pilot should answer the second question. When it is used to answer the first, it runs for months, the goalposts move, and by the time the technology is proven nobody remembers what “success” was supposed to mean.
Six reasons pilots stall
1. Nobody measured the baseline
“Scrap went down” means nothing without a “before”. Too many pilots start with the system switched on and the baseline reconstructed later from memory, last year’s budget or a quality report that measured something slightly different. A baseline has to be measured with the same method, on the same line, over a long enough period to capture normal variation. It has to be measured before the new system starts influencing the result.
2. Success criteria were never written down
“We’ll know it when we see it” is how pilots become permanent. Success criteria need to be specific enough that two people reading them independently would reach the same verdict:
- The metric, for example escaped defects per 10,000 units or unplanned downtime hours on monitored assets.
- The threshold that counts as success, and the one that counts as failure.
- How the metric is measured, by whom and from which system.
- The duration and the minimum sample size.
- Who signs the result off.
Write these before the pilot starts, and get them signed by the person who will approve the rollout budget.
3. The pilot ran on hardware that can’t go to production
A laptop with a gaming GPU, a USB webcam on a magnetic mount and a cloud inference endpoint can prove that a model can see a defect. None of it survives a washdown, a 50 °C enclosure or a security review. When the pilot hardware is not the production hardware, scaling means re-engineering: new optics, new compute, new mounting, new latency. The validation you did in the pilot no longer applies, because you are now validating a different system.
4. IT/OT security saw it last
The security review often arrives in week ten, when the pilot is “done” and someone asks how the device connects to the network. Then come the questions nobody planned for. Which zone does it sit in? What crosses the conduit between that zone and the business network? Who has remote access, and how are updates signed? The answers can force a redesign or stop the project entirely. Bring the IT/OT and security leads into the kickoff, show them the network diagram in week one, and design to the plant’s zones and conduits from the start. Our IEC 62443 hardening checklist covers what they are likely to ask.
5. Nobody owns the model after go-live
A model is not a finished product. Light changes, suppliers change and tooling wears, so the model drifts. We have written about why inspection models get worse over time. During a pilot, the vendor’s engineers watch the model closely. After go-live, if nobody is named to watch confidence trends, review borderline images and approve retraining, the model slowly degrades until operators stop trusting it. Then it gets bypassed. Ownership needs a person, a budget line for retraining and a drift threshold that triggers action.
6. Every site became a custom project
The first line gets a bespoke bracket, a hand-tuned PLC handshake, a labeling convention invented on the spot and a dashboard built for one plant manager. The second site starts from scratch, and so does the third. At that point a fleet costs the same per site as the pilot did, and the business case collapses. Site-specific work should be limited to what physics and process actually demand, such as lighting geometry, the defect catalogue and asset selection. Everything else should be standard.
Symptoms, causes and fixes
| What you see at the end of the pilot | Root cause | Fix in the pilot design |
|---|---|---|
| “It seems to help” but nobody can quantify it | No baseline | Measure for several weeks before switching on |
| Endless extension, moving goalposts | No written success criteria | Signed criteria, fixed duration |
| Rollout quote far above pilot cost | Pilot on non-production hardware | Pilot on the deployable kit |
| Rollout blocked by IT | Late security review | Security lead at kickoff |
| Good results fade within a year | No model owner | Named owner, drift monitoring, retraining budget |
| Site two costs as much as site one | Per-site customization | Standard kit and integration pattern |
A scale-ready pilot checklist
Use this before any pilot starts, with any vendor, including us.
Before the pilot
- A one-paragraph business question the pilot will answer, and the rollout decision that depends on it.
- A baseline measured on the target line, using the same method that will be used at the end.
- Written success and failure criteria, signed by the budget owner.
- Hardware, enclosure and mounting that would be used in production.
- A network diagram reviewed by IT/OT security, showing zones, conduits and remote access.
- The plant-side model owner named, with the time they will spend on it agreed.
During the pilot
- Weekly review of the agreed metric against the baseline, not of anecdotes.
- A log of every manual intervention, threshold change and model update.
- Operators and maintenance staff trained as they would be at rollout, not by the vendor’s engineers standing beside the line.
At the end
- A written result against the criteria, with the raw data available to both sides.
- A rollout plan listing what is standard and what is site-specific, with a cost per additional site.
- A support plan covering drift monitoring, retraining, hardware replacement and who answers the phone at 2 a.m.
How we structure it
Our Proof of Value is built around this checklist. It runs for 6 weeks on your line at a fixed $24,000, covering one inspection station or up to 25 monitored assets, with loaner hardware included and success criteria agreed in writing before we start. The exit is a measured result against those criteria, not a demo. If you go ahead, 50% of the fee is credited to a subscription. Where the defect itself is in doubt, a 3-week Feasibility Sprint on the bench comes first, so the Proof of Value tests value rather than feasibility.
After go-live, Perflens Care covers the ownership gap. Every tier includes drift monitoring and a monthly health report, and retraining runs from twice a year to continuously, depending on the tier. The plant still owns the model, the data and the decisions. Care makes sure someone is watching.
The question to ask before you sign
Before any pilot, ask one question: if this works, what happens on Monday? If the answer involves new hardware, a fresh security review, a new owner or a rebuild for the next site, the pilot is a science project. That can be worth doing, but call it what it is and budget for it. If the answer is “we order the next five stations”, you have designed a first deployment, and it is far more likely to become a fleet.