A vision-language model (VLM) can look at a rejected pouch and tell an operator, in a plain sentence, that product is trapped in the top seal about a third of the way in from the left edge. It can find every similar reject from last month in seconds. What it should not do is decide whether the pouch gets rejected in the first place.
That is the line we draw on every project. VLMs are good at the language around inspection: search, triage, explanation, reporting and onboarding new defect types. They are poor at the decision itself. The pattern that works is a deterministic detector that makes every pass/fail call, and a VLM beside it that explains, cites the frames it is talking about, and never touches the reject gate.
What a VLM is actually doing
A VLM pairs an image encoder with a language model. The encoder turns the image into a sequence of visual tokens; the language model then generates text one token at a time, conditioned on those tokens and on your prompt. Two consequences matter on a factory floor.
First, the output is generated, not computed. The model produces the most plausible continuation of the prompt given the image, and plausible is not the same as true. Second, the model is open-vocabulary: you can ask about anything, in any words, without training a class for it. That flexibility is the whole appeal, and it is also why the output is hard to pin down.
Where VLMs genuinely help
Searching inspection history
A busy inspection cell stores thousands of reject images a week, each tagged with a class, a confidence and a timestamp. Quality engineers rarely have time to scroll through them. A VLM turns that archive into something you can question: “show me seal rejects from line 3 this week where the wrinkle is near the tear notch.” The answer is a set of frames and records that an engineer can check in a minute, instead of an afternoon of filtering spreadsheets.
Triage of rejects and alerts
When the reject rate jumps on a Tuesday afternoon, the first question is whether it is one problem or several. A VLM can group recent rejects by what they look like, describe each group in a sentence, and point out the group that doesn’t resemble any known defect class. It is a first pass for a person, not a verdict, but it turns an hour of scrolling into ten minutes of checking.
Explaining rejects to the people who handle them
A detector outputs seal_contamination 0.94. An operator needs to know what to look at and what to do about it. Given the frame, the crop, the detector’s class and the matching defect catalogue entry, a VLM can write: “Product trapped in the top seal, left third of the pouch, highlighted in frame 18432. Catalogue entry SC-2 suggests checking the sealing jaws for product buildup.” That sentence is often what makes operators trust a system rather than work around it.
Drafting shift reports
An end-of-shift report is mostly numbers the system already has (reject counts by class, top events, stoppages) plus a few sentences of context. A VLM can draft it from structured data and a handful of representative frames, and the shift lead edits rather than writes. The numbers are inserted directly from the inspection database, so the model writes prose around facts instead of producing the facts.
Onboarding a new defect type
When a new defect appears, you usually have a description long before you have a labeled dataset: “a thin white line along the bottom crimp.” A VLM can search the archive for frames that match that description and hand labelers a candidate set to confirm. That speeds up collecting training data for the deterministic model. It does not replace training one.
Where they fail
| Failure mode | What it looks like on the line | Why it rules out pass/fail |
|---|---|---|
| Variable output | Same frame, slightly different prompt, different answer | Decisions can’t be repeated or validated |
| Hallucinated defects | A confident description of a scratch that isn’t there | False rejects that arrive with a persuasive explanation |
| Lost fine detail | A 4-pixel defect vanishes when the frame is downscaled for the encoder | Escapes on exactly the defects that matter |
| Latency | Hundreds of milliseconds to several seconds per answer | Too slow for a reject gate at line speed |
| Prompt sensitivity | Rewording a question changes the verdict | The real specification becomes whatever the prompt says today |
Output you can’t repeat
With sampling turned on, a VLM gives different answers to the same question by design. Turning sampling off helps, but it does not make the system repeatable in the sense a quality engineer means. Small changes in the prompt, the crop, the quantization or the inference software can still change the output. An inspection decision has to be the same tomorrow as today for the same part, and you have to be able to prove that to an auditor.
Hallucinated defects
Language models are trained to produce fluent, plausible text. Asked “is there a scratch on this part?”, a VLM leans toward describing one, especially on textured or reflective surfaces where something scratch-like is almost always visible. Worse, its mistakes are persuasive. A false reject from a classifier is a number on a screen; a false reject from a VLM comes with a paragraph explaining why it is right.
Resolution and fine detail
Most VLMs process images at a limited resolution, often a few hundred pixels per side or a fixed budget of image tokens spread across tiles. A 5 MP inspection frame is heavily downscaled or chopped up before the language model sees it. A defect that spans four pixels in the original frame, the scale a purpose-built detector is designed around, may not survive at all.
Latency
Generating an answer token by token takes hundreds of milliseconds to several seconds on edge hardware, depending on model size and answer length. At 300 parts per minute you have 200 ms per part, and the reject decision has to arrive well before the part reaches the gate. A trained detector on the same hardware decides in tens of milliseconds, every time.
The hybrid pattern: the detector decides, the VLM explains
The architecture we use separates the decision path from the explanation path completely.
- The detector decides. A deterministic, versioned model (a supervised classifier, an anomaly detector or both, as described in anomaly detection vs supervised classification) produces the verdict, class, region and confidence. Its output goes straight to the PLC.
- The record is written. The frame, the crop, the heat map and the detector output are stored together with the model version.
- The VLM is asked, asynchronously. Off the critical path, the VLM receives the record plus retrieved context: the defect catalogue entry, the inspection spec and recent similar rejects.
- The answer must cite. Every statement in an explanation references a frame ID, a record or a catalogue entry. Statements without a citation are dropped before anything is displayed.
- A person acknowledges. Explanations and reports stay drafts until someone signs them off.
Citations are the guardrail that matters most
Requiring citations changes how the model behaves and, more importantly, how people use it. An operator who reads “see frame 18432” clicks through and looks. If the frame doesn’t show what the sentence claims, the error is visible in seconds. An answer without citations asks to be trusted; an answer with citations asks to be checked.
Disagreement is a signal, not an override
Sometimes the VLM will describe a rejected part as clean, or point out something odd on a part the detector passed. Neither changes the verdict. Both go to a review queue. Over a few weeks, that queue becomes a good source of borderline images for the next retraining round: the frames where two very different models disagree are the ones most worth a quality engineer’s time.
Running it on-device
We run VLMs at the edge for the same reasons we run detectors there. Production images reveal product designs, process settings and sometimes people, and many plants will not send them to a third-party service at all. An on-device model keeps working when the WAN link doesn’t, and its behavior can’t change under you because a provider updated a hosted model overnight. Our note on air-gapped inspection covers the wider case.
Sizing is different from sizing a detector. A VLM is limited mostly by memory capacity and bandwidth rather than peak TOPS, and a quantized model with a few billion parameters is the practical range for a line-side box today. In our Private Vision-Language Copilot, the VLM runs on a Gatepoint G8 next to the inspection models, and the inspection models always have priority: if the copilot is busy, explanations wait in a queue and inspection never does. Our edge compute guide goes deeper on hardware.
Guardrails we put around every copilot
- Read-only access. The VLM can read images and records. It has no write path to the PLC, the reject logic or the model registry.
- Retrieval, not recall. Answers are grounded in the defect catalogue, the spec and the inspection database, retrieved when the question is asked, not in whatever the model absorbed during pre-training.
- Structured output for reports. Shift reports are generated into a fixed template, and every count and rate is inserted from the database rather than written by the model.
- Versioned together. The VLM weights, quantization and system prompt are versioned as one unit and recorded in a model card, like any other model.
- A fixed evaluation set. A set of questions with known answers, drawn from your own lines, is re-run before every update. A regression blocks the release.
- Permission to say “I can’t tell.” The prompt and the evaluation both reward “not determinable from this frame” over a confident guess.
Where to start
If you are considering a VLM in a plant, start where mistakes are cheap and time saved is easy to measure: search over existing inspection history and shift-report drafting. Measure how long quality engineers spend answering “what happened on line 3 last night?” before and after. Leave pass/fail to the inspection models you can validate, and let the VLM do the explaining.