Wrappers die in production: the case for orchestration

A demo is one model call. Production is a system. Most of the pain in shipping AI comes from not respecting that gap early enough.

The demo is easy. You wrap a good model in a nice interface, it does something impressive on stage, everyone claps. Then you point ten thousand real users at it, and by week three the cracks show. The bill is bigger than the plan. The slow responses drive people away. The edge cases you never saw in testing are now support tickets. The thing that looked like a product was a single API call wearing a costume.

I have shipped enough AI to know the wrapper does not survive contact with production. What survives is orchestration: treating the model as one component in a system you actually design.

Why one model call is not enough

Route everything through your best, biggest model and three things break at once. It costs too much, because you are paying premium prices for questions a small model could answer. It is too slow, because the biggest model is rarely the fastest. And it is fragile, because when that one call fails or hallucinates, you have no plan B and the user gets the raw failure.

The fix is not a better prompt. It is a better system around the prompt.

What you actually build

Routing. Not every request needs your heaviest model. Send the easy ones to a small fast model and reserve the expensive one for the hard cases. Most of your volume is easy, so routing is where the economics get sane.

Caching. A surprising share of requests are near-duplicates of ones you have already answered. Semantic caching turns those into instant, free responses. It is the cheapest performance win available and most teams add it far too late.

Guardrails. Validate the model's output before it reaches the user, not after. Check it against the rules the model does not reliably follow on its own. The model proposes; your system decides what actually ships to the person.

Fallbacks. When a call fails, times out, or comes back low-confidence, degrade on purpose. A slightly worse answer delivered reliably beats a great answer that shows up half the time.

The metrics nobody tracks early enough

When I scope an AI product now, I care about numbers most teams do not look at until they are already bleeding.

Blended cost per active user, not cost per call, because that is the number that decides whether the unit economics work at scale. Fallback rate, because it tells you how often the smart path is failing quietly. Latency at p90 and p99, not the average, because the average hides the slow tail that users actually feel. And containment: how often the system finishes the job without handing it to a human.

These are product metrics, not engineering trivia. They decide whether the thing is a business or a science project.

What this does to the PM job

Speccing an AI feature is not like speccing a normal one. You are not writing "when the user clicks X, Y happens." You are owning the behavior and the economics of a probabilistic system: what it routes where, what it caches, how it fails, what it costs per user, and where a human stays in the loop.

This is also why, on the products that mattered most, we built the system in-house instead of renting a wrapper. Not for the bragging rights. For control. When the model is one part you own inside a system you designed, you can tune the routing, own the guardrails, and move the cost and latency curves yourself. When it is someone else's black box, you are stuck with their tradeoffs and their bill.

The demo earns the meeting. The system earns the retention. Build for the second one from the start.