Putting an LLM into production is mostly the boring parts

The first time I wired an LLM into a real product, the demo took an afternoon and I felt like a genius. Then it went in front of actual users and I spent the next three weeks on everything the demo had let me ignore. That gap, between "it works when I try it" and "it works when a thousand people try it," is the whole job. Here is where the time actually goes.

The demo hides all the hard cases

In a demo you type a sensible question and get a sensible answer. Real users paste in half a sentence, a wall of text, another language, or something the model was never meant to handle. The model does not say "I am not sure." It answers anyway, confidently, and sometimes it is wrong in a way that looks completely reasonable.

So the first real work is not the prompt. It is deciding what happens when the input is weird or the answer is low quality. What do you show the user? Do you fall back to something simpler? Do you just say you could not help this time? A good "I do not know" beats a confident wrong answer every single time, and you have to build that on purpose.

You cannot improve what you cannot measure

Early on I was tweaking prompts by feel. I would change a word, try three examples, decide it was better, and move on. That is not engineering, that is vibes.

What actually helped was boring: collect a set of real questions, write down what a good answer looks like for each, and run the whole set every time you change something. Now when I change a prompt I can see that it fixed four cases and broke two, instead of guessing. It is not fancy and it is not exciting, but it is the difference between improving the feature and just moving the bugs around.

Cost and speed are features, not afterthoughts

Every call to a big model costs money and takes time. On a demo you never notice. In production, where the feature runs thousands of times a day, both add up fast.

A few things that consistently help:

• Use the smallest model that does the job. The biggest model is rarely worth it. Most tasks are fine on something cheaper and faster, and users feel the speed.
• Do not send the model work it does not need. Trim the context. Half the cost is often text you attached out of habit.
• Cache the answers you can. If people ask the same thing, you do not need to pay for it twice.

Keeping it honest

The part I care about most is not letting the thing make things up. For anything where being wrong matters, we ground the model in real data we control and ask it to answer only from that, rather than from whatever it half-remembers. When it does not have the answer, it should say so.

None of this is glamorous. There is no clever prompt that makes it all go away. Shipping an AI feature that people actually trust is mostly measurement, guardrails and knowing what to do when the model is out of its depth. The demo is an afternoon. The rest is the job.

Thinking about adding an AI feature and want it to hold up in the real world? Talk to us and we will be honest about what is worth doing.