Most of the generative AI work we get asked about starts with a chatbot. Someone in the business saw a demo, wired up an API key over a weekend, and now there's a prototype answering questions about internal documents. It works well enough in the demo to get budget approved. Then it goes in front of real users and the questions get weird, the answers get confidently wrong, and nobody can explain why the model said what it said.
The gap between that prototype and something you can run in production is not a model problem. It's an engineering problem. It looks a lot like the gap between a Jupyter notebook and a deployed service: you need evaluation, versioning, monitoring, cost controls, and a clear story about what happens when the system is wrong. This article is about the patterns that close that gap, and about the non-chatbot shapes generative AI takes once you stop thinking of it as a conversation.
Chatbots Are the Least Useful Shape for Most Business Problems
A chat interface puts the burden of prompting on the user. It's open-ended, which means the input space is unbounded, which means your evaluation surface is unbounded too. For a consumer product where exploration is the point, fine. For an internal business process, it's usually the wrong container.
The generative AI features that hold up in production tend to be narrow, embedded, and invisible. Some shapes we keep coming back to:
- Extraction and structuring — turn an invoice, contract, resume, or support email into a validated JSON object that flows into an existing system. The output has a schema, so you can actually test it.
- Classification and routing — triage inbound tickets, tag transactions, flag documents for review. Often a fine-tuned smaller model or even a classical classifier beats an LLM here on cost and latency.
- Drafting with a human in the loop — generate the first version of a reply, a summary, a product description, a code migration. The human approves. Your error tolerance goes way up because a person is the last gate.
- Semantic search and retrieval — no generation at all in some cases. Embeddings plus a good ranker solve more "the AI should answer questions about our docs" requests than people expect.
- Agentic workflows with tight tool scope — the model chooses among five or six well-defined actions with real validation on each. Not an autonomous agent with shell access.
Notice what these have in common: constrained inputs, structured outputs, or a human checkpoint. That's what makes them testable. If you can't describe what a correct output looks like, you can't ship it responsibly.
Evaluation Is the Thing Teams Skip and Regret
Nearly every stalled generative AI project we've been brought in to unstick has the same missing piece: there is no evaluation set. The team is tuning prompts by vibes. Someone changes a prompt to fix one complaint and silently breaks four things nobody noticed.
Build the eval set before you optimize anything. It does not need to be large. A hundred well-chosen examples with expected outputs will tell you more than a thousand random ones.
What Goes in an Eval Set
- Real inputs from your domain, not synthetic ones you wrote to make the model look good.
- The hard cases: ambiguous documents, missing fields, multiple languages, adversarial phrasing, the customer who writes in all caps.
- Cases where the correct answer is "I don't know" or "escalate to a human." Refusal behavior needs testing as much as answer behavior.
- Regression cases — every production bug becomes a permanent test.
How to Score
Scoring depends on the output shape, and this is where teams overcomplicate things. Use the cheapest reliable method available:
- Deterministic checks first. Does it parse as valid JSON? Does the schema validate? Is the extracted date a real date? Do the numbers sum correctly? A large share of failures are caught here for free.
- Reference comparison where you have ground truth — exact match, field-level accuracy, or fuzzy string matching for names and addresses.
- LLM-as-judge for open-ended text, but only after you've measured the judge against human ratings on a sample. An unvalidated judge is just a second source of unmeasured error.
- Human review on a rotating sample, always. Budget for it. A few hours a week of someone competent reading outputs catches things no automated metric will.
Then wire the eval into CI. Prompt changes, model version bumps, retrieval config changes — all of it runs against the suite before merge. This single practice changes the character of the project. Suddenly you can iterate quickly because you can see what you broke.
Retrieval Quality Beats Model Quality Most of the Time
When a RAG system gives bad answers, the instinct is to switch to a bigger model. Nine times out of ten the problem is upstream: the right chunk was never retrieved, so the model had nothing to work with. Debug retrieval first, in isolation, before you touch generation.
Practical things that move the needle more than model choice:
- Chunking that respects document structure — sections, clauses, table rows — instead of fixed character windows that cut sentences in half.
- Hybrid search. Dense embeddings miss exact identifiers, product codes, and rare terms. Combine with keyword search and fuse the rankings.
- A reranker over the top 50 candidates. This is often the single highest-leverage addition to a mediocre RAG pipeline.
- Metadata filtering so a query about the 2024 policy never sees the 2019 document.
- Measuring retrieval separately — recall@k on a labelled query set. If the right chunk isn't in the top k, no prompt engineering will save you.
There's also a data hygiene point that isn't glamorous. Stale documents, duplicated policies, and contradictory internal wikis produce contradictory answers, and users blame the AI. Part of any serious deployment is deciding what the source of truth actually is, and who owns keeping it current.
Governance That Engineers Will Actually Follow
Governance gets a bad name because it often arrives as a slide deck. The version that works is a set of technical controls and a short, specific policy that fits on one page.
The controls we'd consider table stakes for any system handling real business data:
- A model gateway. All calls go through one internal service, never directly from application code to a vendor SDK. This gives you a single place for auth, rate limits, logging, cost attribution, PII redaction, and switching providers without a redeploy.
- Prompt and config versioning in source control. Prompts are code. They get reviewed, they get tagged, and every logged inference records which version produced it.
- Data classification rules — what categories of data may leave your network, which must go to a self-hosted or region-locked model, and what gets redacted before the call. Enforce this in the gateway, not in a wiki page.
- Permission-aware retrieval. The retriever must respect the calling user's access rights. A search index that ignores ACLs is a data leak with extra steps, and it's one of the most common serious mistakes we see.
- Output guardrails appropriate to risk — schema validation, PII scanning on outputs, injection detection on retrieved content, and refusal paths for out-of-scope requests.
- A documented human-review threshold. Below a confidence score, or for certain categories, the output goes to a person. Write down where that line is and why.
If you operate in the EU, the AI Act's risk tiers will shape how much documentation and human oversight you need; sector rules like HIPAA or financial services regulation may add more. We're engineers, not your counsel, so get proper legal review on classification. But build the audit trail from day one either way. Retrofitting traceability onto a live system is miserable work.
Observability: Logs Are Not Enough
Traditional APM tells you a request took 4 seconds and returned 200. For an LLM feature that's almost useless. You need to see inside the call.
Instrument every inference as a trace with the full picture: the user's input, the resolved prompt, retrieved context with document IDs and scores, tool calls and their results, the raw completion, token counts, latency per step, model and prompt version, and cost. Sample or redact sensitive fields according to your data policy, but capture the shape of every call.
Then watch for the failure modes that are specific to this kind of system:
- Drift in input distribution — users start asking about something your retrieval corpus doesn't cover, and quality quietly drops.
- Rising refusal or fallback rates, which usually signal a retrieval or prompt problem rather than genuine out-of-scope traffic.
- Cost per resolved task, not cost per token. A cheap model that needs three retries is not cheap.
- p95 latency including retrieval and reranking, because that's what the user feels.
- Silent provider changes. Hosted models get updated. Your eval suite running on a schedule is how you find out before your users do.
Feed user signals back in: thumbs down, edits to generated drafts, escalations to human agents. An edited draft is a labelled training example handed to you for free. Systems that capture this get better every month. Systems that don't stay exactly as good as they were on launch day.
Cost, Latency, and Knowing When Not to Use an LLM
An honest word here. A meaningful share of the generative AI requests we receive are better served by something simpler. If the task is a deterministic rules problem, write the rules. If you have thousands of labelled examples and a stable label set, a fine-tuned small classifier will be faster, cheaper, more accurate, and easier to explain than any prompt. If you need exact numbers, use a database and let the model only translate the question into a query.
When an LLM is genuinely the right tool, the levers that matter in production are mundane: cache aggressively at the semantic level, route easy cases to small models and hard cases to large ones, keep prompts short because context length drives both cost and latency, and batch anything that isn't user-facing. Start on a hosted API to validate the use case. Consider self-hosting an open-weight model only when volume, data residency, or latency requirements justify the operational load — and be honest that GPU capacity planning is a real ongoing cost, not a one-time saving.
A Sequence That Tends to Work
- Pick one workflow with a measurable business metric — hours saved, tickets resolved without escalation, cycle time on a document process. Not "deploy AI."
- Build the eval set from real data before writing a prompt.
- Ship the thinnest version behind a flag to a small internal group, with a human checkpoint on every output.
- Instrument fully from the first day of that pilot.
- Read the traces. Fix the top failure mode. Repeat until the metric moves.
- Loosen the human checkpoint only where the data says you can, category by category.
- Then widen the audience, and only then consider the next workflow.
None of this is exotic. It's ordinary software discipline applied to a probabilistic component, and the teams who treat it that way get to production while the teams chasing a better prompt are still in the demo phase. If you want to see the kinds of systems we've built, our recent work is a reasonable place to start, and our AI and ML engineering practice covers how we usually structure these engagements. For longer programmes, an embedded team alongside your own engineers is often the faster route than a fixed-scope build.
If you have a generative AI prototype that works in a demo and stalls everywhere else, we're happy to look at it with you and give you a straight answer about what it needs — including if that answer is "less AI than you think." Get in touch and tell us what you're trying to build.
