Research

Your agent is not confused. It was never told.

Most pilots that die in production do not fail on model quality. They fail because nobody wrote down what the business actually means, and nobody tested whether the machine got it.

By Subhash Trivedi30 July 20264 min read

Part I

The pilot that passed every demo

A finance team we worked with had an analyst agent that demoed beautifully. It answered every question the steering group asked, sourced its numbers, and won the budget. Three weeks after launch it was quietly switched off.

The reason was not the model. Someone asked for revenue by segment and got a number 4% higher than the board pack, because the agent had never been told that intercompany transfers are excluded after the fiscal close. Nothing in the system was wrong. Something in the business had simply never been written down anywhere a machine could read it.

That is the pattern behind most stalled deployments we are called into. Teams debate models, retrieval strategies and prompt frameworks. The actual defect is that the shared meaning of the business lives in people's heads, in half-documented dashboards, and in the exceptions a senior analyst applies without thinking.

Part II

Four ways context breaks

Cold start

You have a hundred candidate use cases and no machine-readable description of what any of your data means. So the first sprint is spent writing definitions from a blank page, which nobody has time for, so it never finishes.

Two teams, one word

Finance and sales both have a correct definition of revenue, and they are different. A single global dictionary forces one of them to be wrong. Left alone, whichever definition the agent happened to absorb becomes policy by accident.

No way to test

Nobody would ship application code with no tests, yet the meaning layer underneath an agent usually ships on a vibe check. Without a scored suite, you find out what the agent misunderstood when a user does.

No owner after launch

Definitions drift the moment a policy or a product changes. If no named person owns the meaning of a term, the agent is accurate on the day it ships and slowly wrong from then on.

Part III

Treat context like code

Every practice that made software delivery reliable applies here, and almost none of it is being applied. Meaning should live in versioned, reviewable units, scoped to a domain rather than to the whole company. Finance's revenue and sales' revenue can both exist, in separate places, each with an owner's name against it.

Tests come from material you already have. Your dashboards, saved queries and board packs are a record of the questions people actually ask and the answers they expect. Turn a few hundred of those into a scored suite, run the agent against it before release, and the vibe check becomes a number you can defend to a CFO.

Then roll out the way you would roll out anything else. Stage it, watch what the agent retrieves as well as what it says, and make corrections cheap. When a user flags a wrong answer, that correction should update the definition once and reach every agent reading from it, not get fixed in one prompt and forgotten in four others.

The division of labour that works: the machine drafts, because it can read every table and thread faster than any team. Domain experts verify and add the exceptions, in plain language, because that knowledge exists nowhere else. Engineers own the pipeline that gets it into production and keeps it tested. Nobody in that arrangement is doing the other's job.

One definition, written properly

termrevenue
domainfinance
meansnet sales after returns, post tax
excludesintercompany transfers
windowfiscal year close
ownergroup controller
tests41 of 47 passing

The exclusion line is the one that killed the pilot in Part I. It took a controller ninety seconds to say out loud and had never been recorded anywhere.

Is your context production ready?

·Every term your agent uses has a written definition a person could argue with.

·Where two teams disagree, both definitions exist and are scoped, rather than one silently winning.

·A named person owns each definition and hears about it when the policy changes.

·You can state a pass rate on real questions, not an impression from a demo.

·You can see which context an answer used, not just the answer.

·A user correction updates the definition once and reaches every agent that reads it.

·Rolling back to last month's definitions is a routine action, not an incident.

Fewer than four of these usually explains a stalled pilot better than any model comparison will.

None of this is glamorous work, and none of it can be bought as a licence. It is the difference between an agent that impresses a steering group and one that a controller is willing to quote in a board pack.

Start with the questions your teams already ask.

Our Discovery work builds the index and the definitions behind it, and scores them against real questions before anything reaches your users.

See Discovery