What it takes to move an AI pilot into day-to-day production

Most AI implementation programs don’t fail in the pilot. They fail in the quiet months after it, when a demo that impressed the steering committee meets real data volumes, real users and a finance team asking what each answer costs.

The research backs this up. MIT’s Project NANDA, in its July 2025 report The GenAI Divide, found that only 5% of custom enterprise AI tools reach production. The authors tied that gap to brittle workflows, weak contextual learning and poor fit with daily operations, not to model quality. Separately, S&P Global Market Intelligence surveyed more than 1,000 organizations across North America and Europe. The average respondent scrapped 46% of its AI proofs of concept before they went live.

So the useful question for a CTO or CDAO is narrower than “does the model work?” Ask instead: what was the pilot quietly relying on that go-live will take away?

Why AI implementation stalls after a good pilot

A pilot is a protected environment. Somebody hand-pulled a clean extract. A data scientist watched every output. Token spend sat on a corporate card nobody reconciled. Twenty friendly users forgave the odd wrong answer, and nothing the model said could move money or touch a customer record.

None of those conditions survive launch. I find it useful to list each subsidy explicitly, then decide who pays for it once the pilot ends. That list becomes the real scope of an AI implementation. It is usually larger than the model work itself.

Subsidy one: data someone cleaned by hand

Pilot data is typically a snapshot. An analyst exported it, fixed the obvious nulls, dropped awkward edge cases and loaded it into a notebook or vector index once. A live system needs that same data delivered continuously, at a known freshness, in a stable shape.

In a February 2025 release, Gartner reported that 63% of organizations either lack or are unsure they have the right data management practices for AI. Its analysts predict that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data.

Three things need to exist before cutover:

• Data contracts on every upstream source. Schema, null rules and refresh cadence get agreed with the owning team. Pipeline tests fail loudly when a contract breaks. An upstream column rename should page an engineer, not silently degrade answers for a week.
• Freshness SLAs for retrieval. If you run retrieval-augmented generation, the index is a derived dataset. Decide how stale it may get and monitor re-embedding lag. Version the embedding model so the index can be rebuilt cleanly when you swap it.
• Permissions that travel with the data. Pilots often index a shared folder through one service account. Once live, a sales rep must not retrieve HR documents just because they were chunked into the same index. Document and row-level access control has to be enforced at query time.

Subsidy two: a human judging every answer

During the pilot, quality gets assessed by people reading outputs and nodding. That works for 200 samples. At 20,000 requests a day it collapses, and it gives you no way to tell whether Tuesday’s prompt tweak helped or hurt.

Evaluation has to behave like a test suite:

1. Build a golden set of a few hundred real inputs with agreed correct outputs. Draw them from the messy middle of your traffic, not the showcase cases.
2. Score every change to prompts, retrieval settings or model versions against that set in CI. A release that drops accuracy on a critical slice does not ship.
3. If a model grades outputs, calibrate it against human reviewers first and recheck that agreement periodically. An uncalibrated judge simply automates optimism.
4. Pin model versions. Hosted providers update and retire models on their own timelines, and an unannounced behavior shift is a regression you did not write.

On live traffic, sample outputs for human review and track retrieval hit rates. Log every trace with its input, retrieved context, output and latency. When a user complains, an engineer should be able to replay that exact request within minutes.

Subsidy three: nobody priced the answers

Pilot budgets hide unit economics, and launch exposes them quickly. In the S&P Global research, cost ranked among the top obstacles respondents cited, alongside data privacy and security risk.

The metric that matters is cost per completed business outcome: per invoice processed, per ticket resolved, per claim triaged. Token price alone tells you little. Consider a workflow that calls a large model four times, retries on malformed JSON and stuffs 12 retrieved chunks into each prompt. That is at least four times the spend of the single call the pilot priced, before counting the larger prompts.

The levers are well understood. Someone still has to own them:

• Route simple requests to smaller, cheaper models and reserve frontier models for hard cases.
• Cache repeated context and deterministic sub-results.
• Trim retrieval to what measurably improves accuracy on the golden set.
• Set a p95 latency budget per step, since users abandon a slow assistant even when it is right.

Put cost per outcome on the same dashboard as accuracy. When finance asks, the figure should already exist.

Subsidy four: forgiving users and no blast radius

Pilot users volunteered. They tolerate errors because they want the project to succeed. Everyday users did not sign up for an experiment, and some will break the system, deliberately or by accident.

Agentic designs raise the stakes because the model now takes actions instead of drafting text. Gartner predicted in June 2025 that over 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls.

Before any AI system touches a live process, I would expect to see these controls:

• Confidence-based routing. Low-confidence outputs go to a human queue. Track the escalation rate; if it exceeds what operations can staff, the use case isn’t ready.
• Scoped tool permissions. An agent that reads the ERP should not post journal entries unless that is its explicit job, and then only within limits.
• Shadow mode before cutover. Run the system alongside the existing process for a few weeks, compare decisions and investigate every disagreement.
• A rehearsed rollback. A feature flag that reverts to the manual path within minutes, tested at least once before launch.
• An audit trail that lets compliance reconstruct who or what made a decision, and on what evidence.

Subsidy five: the workflow never changed

This subsidy is organizational, which makes it the hardest to fix with engineering. Many pilots bolt a model onto an existing process. Nobody redesigns the steps around it, so the output lands in an inbox and people keep working the old way.

McKinsey’s State of AI survey, published in November 2025, found only 39% of respondents attribute any EBIT impact to AI. The roughly 6% it classes as high performers were nearly three times as likely as others to have fundamentally redesigned workflows when deploying AI.

Take accounts payable as an example. A pilot that extracts invoice fields and emails them to a clerk saves minutes. An AI implementation that posts matched invoices straight to the ledger, and sends only exceptions to the clerk, changes headcount planning and close timelines. The model can be identical in both cases. Only the second one shows up in EBIT.

Redesign means deciding, step by step, what the model does, what a person verifies and what disappears entirely. It also means naming owners. A live AI system needs an on-call rotation, a business owner accountable for the outcome metric and a defined path for users to flag bad answers. If a model failure at 2 a.m. has no pager attached, you are still running a pilot, whatever the launch email said.

A go-live checklist for AI implementation

Before a pilot graduates, ask the team to answer these in writing:

• Which upstream sources feed this system, and does each have a contract and an owner?
• What is our golden-set accuracy today, and what score blocks a release?
• What does one completed outcome cost, and where is the ceiling?
• What happens when confidence is low, and who staffs that queue?
• How fast can we roll back, and when did we last test it?
• Which workflow steps changed, and who owns the business metric?

A team that answers all six with specifics is ready. A team that answers three has a pilot with a production label on it.

Where to start

Pick the AI pilot your organization is proudest of. Write down every subsidy it currently relies on. Next to each one, write the name of the person who will carry that cost after launch. The blank names are your real implementation backlog.