Practical guides

How to move an AI pilot into production

A pilot proves a model can do the thing once. Production is a different claim: that it keeps doing it, for everyone, on the worst day of the quarter.

By Subhash Trivedi3 August 20268 min read

The most expensive place to be with AI is a successful pilot. The demonstration worked, the enthusiasm is real, and the assumption forms that what remains is deployment. Then six months pass and it is still a pilot.

The gap is rarely technical. A pilot is a controlled experiment: chosen inputs, forgiving users, no obligation to be available at nine on a Monday. Production is an operational commitment with a name attached to it. Most of the work between the two is work the pilot was designed not to do.

What follows is what that work consists of, and the order that tends to hold.

What a pilot deliberately leaves out

None of these omissions is a mistake. A pilot that carried them would not be a pilot. But they all come due at once.

The unhappy path

Pilots are shown the cases they handle. Production meets the malformed document, the ambiguous request, the record that predates the current system.

Real load

Consumption pricing is cheap at pilot volume. Cost per decision at full volume is a different number, and often a different decision.

Ownership

A pilot belongs to whoever was curious. Production has to belong to the person accountable for the workflow it changes.

Failure behaviour

What the system does when it is unsure is more consequential than what it does when it is confident.

The six things production adds

An evaluation set that reflects your business

A held-out set of real cases with agreed correct answers, including the hard and rare ones. This is the asset that makes every later change safe, and the one most teams postpone. It should be built before the system is promoted, not after the first incident.

A defined service level

Latency, availability, and quality thresholds, written down. Google's SRE guidance on service level objectives predates this technology and still applies to it: an unstated target cannot be missed, which means it also cannot be met.

A path for the cases it should not handle

Escalation, human review, or refusal. A system that answers everything is not more useful than one that knows its boundary. It is harder to trust.

Monitoring on outputs, not just uptime

The failure mode that matters is quality drift while the service stays green. Track answer quality, refusal rate, and cost per decision on the same dashboard as availability.

Security review of the specific surface

Prompt injection, tool permissions, and data exfiltration through retrieval are not covered by an ordinary application review. The OWASP Top 10 for LLM applications is a reasonable checklist to start from.

A named owner and a stopping condition

One person accountable for the number the system is supposed to move, and an agreed condition under which it is switched off. Both should be settled while everyone is still calm.

Bolting the model on is what fails

The pattern that separates working systems from stalled ones is not model choice. It is whether the process was redesigned around the new capability, or the capability was laid on top of the existing process.

A model that drafts a document, inside a workflow that still requires the same three approvals afterwards, has moved effort rather than removed it. People feel faster. Cycle time does not change. This is the mechanism behind the finding that around 95% of generative AI pilots deliver no measurable impact on the profit and loss statement.

Promotion to production is a decision about the process, not about the model. If nothing upstream or downstream changes, nothing measurable will either.

The uncomfortable version of this: the redesign usually has to be agreed before the build, because it determines what the system needs to do. We wrote about the diagnostic that produces that agreement in discovery.

The order that tends to hold

Teams get into trouble by running these in parallel and discovering late that step four invalidates step one.

Fix the measurement first

Establish the current cost or cycle time of the workflow, using data that already exists. Without a baseline there is no way to prove the change, and the argument about whether it worked will be settled by whoever is most senior in the room.

Build the evaluation set second

Before optimising anything. It defines what better means, and it will outlive every model you use. Model evaluation is a discipline, not a stage.

Then narrow the scope

Promote the smallest version that changes the number. Ninety per cent of a workflow at high reliability beats all of it at moderate reliability, because the remaining ten per cent can be routed to a person.

Then instrument, then load-test, then hand over

In that order. Handover is not documentation. It is someone on your side making a change to the system, unaided, while the original team watches.

When not to promote a pilot

Some pilots should be stopped, and the discipline to stop them is what makes the rest credible.

Stop if the value only appears when you assume the process changes and nobody has agreed to change it. Stop if the cost per decision at real volume exceeds the cost of the human decision it replaces. Stop if no one will put their name against the number. Stop if the only measurable outcome is time saved and you cannot say where that time goes. That last one is the most common, and it is the subject of why AI ROI is an organisational problem.

The teams that get systems into production are not the ones with the best models. They are the ones who decided what the system was for, wrote down how they would know, and were willing to switch it off.

Stuck between a working pilot and a live system.

That gap is most of what we do. If you can name the workflow and the number, we can tell you within a fortnight what promotion actually requires.

Start a conversation What we build