Synthetic defect images are useful for one specific job: filling gaps in real data that you can name. That might be a defect you have seen six times, a crack that has only ever appeared on one side of the part, or a product that hasn’t run yet. Used that way, and judged only on real images, synthetic data can make a model noticeably better on rare classes. Used as a substitute for collecting real data, it teaches a model to recognize the generator rather than the defect.
The rules we work by are short. Generate for a named gap. Keep validation and test sets entirely real. Measure the effect with ratio experiments. Write all of it down in the model card. This article explains why each rule exists.
Why teams reach for synthetic data
A good production line makes very few defects, which is the point. Supervised models learn from examples, though, and examples are scarce exactly where they matter:
- Rare defects. A critical crack that appears a handful of times a year.
- Class imbalance. Thousands of images of one cosmetic blemish and a dozen of the defect that causes recalls.
- Missing variation. Every real example of a defect sits in the same position, orientation or SKU.
- New products. Nothing to photograph until the line runs.
- Defects you can’t safely create. Contamination, some seal failures, or damage that means destroying expensive parts.
Two families of methods
Procedural generation
Procedural methods build defects from explicit rules. The simplest is cut-and-paste: take a real defect patch and blend it onto good images at new positions, scales and rotations. More elaborate versions draw parametric scratches and cracks, or render the part from CAD with physically based materials while randomizing lighting, texture and camera pose, a technique known as domain randomization. The strengths are control and exact labels, because you know precisely which pixels belong to the defect. The weakness is that procedural defects tend to look procedural.
Generative models
Generative methods, including GANs, diffusion models and mask-conditioned inpainting, learn what defects look like from examples and produce new ones. A diffusion model fine-tuned on a few dozen real images of a defect can produce varied, realistic examples, and inpainting can place a defect inside a chosen region of a good image so the mask doubles as the label. The weaknesses are subtler. A generator can copy its training images almost exactly, invent features no real defect has, or draw a defect that spills outside the mask you asked for.
| Method | Best for | Main risk |
|---|---|---|
| Cut-and-paste with blending | New positions and orientations of a known defect | Blending seams that the model learns instead of the defect |
| Parametric drawing | Scratches, cracks and simple geometric defects | Unrealistically uniform shape and contrast |
| CAD rendering with domain randomization | New products, 3D defects, varied poses | A large gap between rendered and real images |
| Fine-tuned diffusion or GAN | Realistic texture and appearance variation | Memorized training examples; plausible but wrong features |
| Mask-conditioned inpainting | Defects in specific zones with known labels | Defect drifting outside its mask |
Where synthetic data goes wrong
The domain gap
Real images carry the fingerprint of a particular camera, lens, light and compression setting: sensor noise, slight blur, vignetting, compression artifacts. Synthetic images carry a different fingerprint. If the gap is large, part of what the model learns is “real versus synthetic” rather than “good versus defective”.
A quick diagnostic is to train a small classifier to tell your real images from your synthetic ones. If it succeeds easily, so can your defect model. Close the gap by passing synthetic images through the same degradations as real ones (matched noise, blur and compression) and by rendering with the actual lighting geometry of the cell rather than a default.
Label leakage
Label leakage happens when something other than the defect predicts the label. With synthetic data it is easy to create by accident:
- every synthetic image is defective and every real image is good, so “is it synthetic?” answers “is it defective?”;
- pasted patches leave halos or seams that appear only on defective images;
- generated defects always sit near the center of the frame, or always on one background;
- the generator was fine-tuned on real defect images that later appear in the test set, so the test measures memory rather than generalization.
The fixes are mechanical. Run the same pipeline on good images too, pasting good patches onto good parts so seams carry no information. Randomize placement within the zones where defects really occur. And split your real data into training, validation and test sets before you build any generator, so no test image ever conditions it.
The “too clean” problem
Synthetic defects tend to be textbook examples: high contrast, centered, isolated, one per image, on a spotless part. Real defects are faint, partly hidden at an edge, mixed in with dust, oil and mold lines, and often close to the acceptance limit. A model trained on the clean version gets very good at obvious defects and no better at borderline ones, and borderline defects are what your inspection spec is really about.
The same applies to good images. If synthetic good parts are cleaner than real ones, normal texture on the line starts to look anomalous, and false rejects climb.
Rules we work by
Validate only on real data
Validation and test sets are real images from production, labeled by your quality team and frozen before any synthetic data is generated. Synthetic images never appear in either. Report recall per defect class and false-reject rate on real data, because those are the only numbers that predict what happens on the line. Size the real test set so the numbers mean something: with 20 real examples of a defect, a single miss moves measured recall by five points.
Run ratio experiments
The right amount of synthetic data is an empirical question. Hold the real training set fixed, add synthetic images in steps, and measure on the real test set each time. Results for a rare crack class might look like this:
| Synthetic share of crack training images | Recall on real cracks | False-reject rate on real good parts |
|---|---|---|
| 0% (real only) | 71% | 0.4% |
| 25% | 82% | 0.4% |
| 50% | 86% | 0.5% |
| 75% | 85% | 0.9% |
| 90% | 79% | 1.4% |
Hypothetical figures, for illustration only. The shape is typical: gains at first, then a plateau, then decline as the model learns more about the generator than about cracks. Choose a share before the knee, and check that the classes you didn’t augment haven’t become worse.
Document it in the model card
Every model trained with synthetic data should say so plainly. The model card should record:
- the generation method, tool and version, with seeds or configuration;
- which classes were augmented, how many images were generated, and the final ratio;
- which real images were used to build or condition the generator;
- real-only test results with and without synthetic data;
- known limits, meaning the defect sizes, positions or variants the synthetic data did not cover.
When the model drifts a year later, as described in why your inspection model got worse in July, this record lets someone work out whether synthetic data is part of the problem.
When not to use it
Synthetic data is the wrong tool when:
- you have no real examples of the defect to validate against, because you would be measuring the generator against itself;
- the defect depends on physics the generator can’t reproduce, such as translucency, depth, or how light scatters inside a material;
- collecting real examples is cheap, for instance by seeding known defects into a production run or pulling retained rejects from the quality archive;
- the real problem is that the defect has poor contrast, which is a lighting problem, not a data problem;
- defects are so varied that a model trained on good parts is a better fit, as discussed in anomaly detection vs supervised classification.
Synthetic data as an engineering tool
Generated images are neither a shortcut nor a gimmick. They are a way to cover specific, named gaps in real data, and they deserve the same rigor as any other change to a model: a hypothesis, a controlled experiment on real images, and a written record. That is how we run synthetic defect data and model training projects, and it is why any synthetic data in our AI visual inspection systems is validated against real samples before it reaches a production model.