The hidden costs of building on top of LLMs

Building on top of a large language model looks cheap and fast at the start. A few API calls, a decent prompt, and you have something that works well enough to demo. That early ease is real, and it is exactly what makes it misleading. The costs that matter do not show up in the first week. They show up later, after you have committed, after you have told customers this is how the product works, after it is too expensive to walk back.

The founders who get burned are not the ones who tried something hard. They are the ones who priced the first call and assumed the rest would be just as cheap. It never is. The beginning is the cheapest part of building on an LLM, and treating it as representative of the whole is how you end up with a feature that works in the demo and quietly loses money in production.

None of what follows is a reason to avoid these models. For the right problem there is nothing else like them. It is a reason to price the whole thing honestly before you build your business on it, so you know what you are actually signing up for.

Why the beginning feels so cheap

The first version of an LLM feature costs almost nothing to build, and that is not an illusion. You write a prompt, you send it to an API, you get back something useful. There is no model to train, no data pipeline to stand up, no infrastructure to run. Work that used to take a team weeks now takes an afternoon. That speed is the whole reason to reach for these models in the first place.

But the demo version and the production version are not the same product wearing different clothes. The demo has one user, who is you, and that user is forgiving. You send a clean input, you read the output yourself, and if it is wrong you shrug and try again. Every cost that matters is absorbed by the fact that you are the only person using it, you are watching every response, and nothing depends on it being right.

Production removes all of those cushions at once. You get many users instead of one, sending inputs you did not anticipate, and nobody is reading the output but the customer. The moment those cushions come off, a set of costs that were hiding behind them arrive together. They were always there. The demo just hid them.

The per-call cost that scales with success

The most obvious hidden cost is the simplest to explain and the easiest to underprice: every interaction with the model costs money, and that cost grows directly with usage. When you are testing, this is invisible. A few cents across an afternoon does not register. When you have real users making constant calls all day, it becomes a serious and growing line in your budget.

The trap is the direction the cost scales. It scales with success. The more people love your feature and the more they use it, the more it costs you to run. That is backwards from how most software works, where serving one more user is nearly free. Here, your most engaged users are your most expensive users, and the growth you wanted most is the thing quietly eroding your margin.

• The heaviest users cost the most to serve, not the least.
• Longer inputs and longer outputs both cost more, and real users produce both.
• Any feature that calls the model more than once per action multiplies the bill.
• Growth makes the problem bigger, not more efficient.

Founders who did not model this discover it the hard way, when a spike in usage they should be celebrating turns into a bill they cannot explain to anyone. The fix is not complicated, but it has to happen before you launch: price the feature at real volume, with real input sizes, and with the number of model calls a single user action actually triggers. Not the volume of one person testing it. If the honest number does not work, you would rather find out now than after you have promised it to customers.

What a single request actually costs

That math surprises people because the mental model is wrong. You picture one request as one call spending a few tokens. A token is just a chunk of text the model reads or writes and gets billed for. That picture holds for a toy prompt. It falls apart for anything real, and it falls apart hardest for the feature founders are most excited to build: a reasoning model driving an agent that calls tools.

A few words first, because they do the work here. A model call is one round trip to the API. Context is everything you hand the model in that call to consider. A tool call is when the model decides it needs to go do something, like run a search or fetch a document, and you run it and feed the result back. Reasoning is the model thinking out loud on its way to an answer. You pay for those thinking tokens too.

Now the part that breaks the naive picture. The model does not remember anything between calls. It has no memory of the last turn. So to keep going, the whole conversation so far has to be re-sent on every call. The system instructions. Every tool definition. The original question. The model's own reasoning from earlier turns. The full result of every tool it already called. One request is not one call. It is a loop, and the context grows on every turn, because last turn's output is this turn's input. You pay for input tokens on every call. So the same early tokens get billed again and again, once per turn, plus everything new piled on top.

Walk one request through. A user asks your product a single question. Behind it, a reasoning model runs an agent loop. It plans, calls a tool, reads the result, calls another tool, reflects, and answers. Round numbers, meant to show the shape and not a vendor's price sheet:

• Fixed overhead re-sent every turn: system prompt plus tool definitions, about 2,000 tokens.
• The user's question: about 500 tokens, carried on every turn after it appears.
• Turn 1: the model reads the overhead plus the question, about 2,500 tokens in. It reasons and calls a search tool, about 500 tokens out. The search returns about 3,000 tokens. Those join the context for good.
• Turn 2: the model re-reads everything so far, about 6,000 tokens in. That is the 2,500, plus its own 500 of reasoning, plus the 3,000-token search result. It reasons and calls a second tool, about 500 tokens out. That tool returns about 2,000 tokens, which also stay.
• Turn 3: the model re-reads it all again, now about 8,500 tokens in. It reasons and calls a third tool, about 500 tokens out. That tool returns about 1,500 tokens.
• Turn 4: the model re-reads the whole pile, about 10,500 tokens in, and writes the answer, about 800 tokens out.

Add up the input tokens across the four turns. Roughly 2,500 plus 6,000 plus 8,500 plus 10,500. That lands near 27,000 tokens in, plus a few thousand tokens of reasoning and output on top. All for one question that felt like a 500-token question.

The tool results are the sneaky part. Every blob a tool hands back becomes context. The search results, the fetched document, the API response. The model re-reads each one on every turn that follows, and you re-pay for it each time. So the naive estimate of one 500-token call is off by roughly fifty times, before a single dollar sign appears.

Now put a price on it. Keep the price illustrative, because the point is the multiple, not the vendor. Say input runs a few dollars per million tokens and output runs several times that. Twenty-seven thousand input tokens is a bit over a cent. A few thousand output and reasoning tokens at the higher rate adds most of another cent. Call it two cents for that one question. Two cents sounds like nothing. That is exactly the problem.

Two cents, times success

Two cents is invisible when you are the only user, asking the thing a handful of times while you build it. That is the demo again, hiding the cost behind the fact that you are alone. Real users are not alone, and they do not ask once.

Take an engaged user who leans on the feature through their workday, triggering that agent loop maybe 25 times a day. Run the numbers:

• One request: about 2 cents.
• One engaged user, 25 requests a day: about 50 cents a day.
• That user over a 30-day month: about 15 dollars.
• 100 engaged users at that rate: about 1,500 dollars a month.

Fifteen hundred dollars a month. From a feature you priced in your head at two cents, serving a hundred people who happen to love it. Push the assumptions a little, heavier users or a busier loop, and the number climbs. Pull them back and it is still hundreds of dollars a month for a hundred users. Either way it is real money, and it arrived entirely from the feature working.

This is the scaling made concrete. The cost scales with success. The more your best users use the feature, the more they cost you. Your best users are your most expensive users. And the per-call estimate already hid a fifty-times multiple inside a single request, before you multiplied by the number of people. Both multipliers are real. They stack. Neither one shows up in the demo.

The cost of non-determinism

A language model does not give you the same answer every time, and that unpredictability is a cost you pay in engineering and in trust. Ordinary software is deterministic. The same input produces the same output, so you can test it once and rely on it forever. A model breaks that assumption. You cannot assume the output will be correct, and you cannot even assume it will be shaped the way you expect, so you have to build around the possibility that it is neither.

That means real work that a conventional system would never need:

• Validation, to check that the output is well-formed before you trust it.
• Fallbacks, for when the model returns something malformed or plainly wrong.
• Retries and guardrails, for the cases that slip through anyway.

All of that is code that exists only because the thing at the center is unreliable. Non-determinism is part of what makes these models powerful. The same flexibility that lets a model handle inputs you never anticipated is what stops it from behaving identically every time. You do not get one without the other. So you pay a tax on every feature you build this way, and the tax is all the scaffolding that has to exist just to handle the model not doing what you asked.

The evaluation cost

Because the output is unreliable and you have many users, you now have a question you did not have in the demo: is this actually working, at scale and over time? Answering it requires evaluation, and evaluation is real work that founders consistently underestimate because in the demo it was free.

In a demo you judge quality by looking. There is one user and a handful of outputs, and your own eyes are the whole quality system. In production you cannot look at everything. Thousands of responses go out that no one on your team will ever read. So you need systems that tell you whether the output is holding up without a human checking each one, which means deciding what "good" even means for your feature and building something that measures it.

Skip this and you are operating blind. You will not know quality has dropped until a customer tells you, and by then the damage is done and you are reacting instead of catching it. The cost of evaluation is the cost of knowing whether the thing you shipped still works. It is not optional if you are serious, and it does not go away, because the model and your users both keep changing under you.

The prompt and glue maintenance cost

The prompts, the logic that assembles them, and the code that stitches the model into your product are not write-once artifacts. They are living things that need maintenance. As your product changes, as new edge cases surface, as the model itself gets updated, all of that scaffolding needs attention. It is easy to think of the model as the hard part and the wiring around it as an afternoon of plumbing. In practice the wiring is most of the work, and it is the part that never stops.

When I built AI agents into Custom1, a tool I built for designing custom protective foam from photos, a large share of the ongoing effort was not the model at all. It was everything around it. The logic that fed it the right information. The checks that verified its output before anything downstream trusted it. The connections into the rest of the system so a result became an actual design and eventually a physical cut. That connective tissue is real software with real maintenance costs, and unlike a demo prompt, it gets more complicated over time as you handle each new case the real world throws at it, not less.

This is the cost that surprises technical founders most, because it does not feel like it belongs to the model. It feels like normal engineering, and it is. That is the point. Building on an LLM does not remove the software you have to write and maintain. It moves it around the model, and there is more of it than the easy start suggested.

The dependency cost

Everything so far is work you do on your side. The next two costs are different. When you build on someone else's model, you inherit their situation, and their situation is not under your control.

• Their pricing, which they can change.
• Their rate limits, which shape what you are able to offer.
• Their availability, which quietly becomes your availability.
• Their decisions about which model versions stay alive and which get retired out from under you.

You have built your product on a foundation you do not own. That dependency does not show up as a cost on any normal day. It shows up the day the foundation moves: a price increase you have to absorb or pass on, a rate limit that caps your growth, an outage on their side that reads as an outage on yours to every customer you have. The more central the model is to your product, the more of your fate you have handed to a company whose priorities are set by their business, not yours. That is not a reason to panic, but it is a real, standing risk that belongs on the ledger with everything else.

The cost of the model changing underneath you

The dependency has one consequence sharp enough to deserve its own name: the model you built on can change, and behavior you were relying on can change with it. The provider ships an update, and something that worked reliably yesterday starts producing different results today. You did not touch your code. The ground under it moved.

This is a strange and specific cost of building on a foundation that itself evolves. Conventional software does not spontaneously behave differently. The library you depended on last year does the same thing this year unless you upgrade it on purpose. A model that someone else keeps improving does not offer you that stability. An update meant to make the model better in general can quietly break the one narrow thing your product leaned on, and you find out from your users. This is exactly why the evaluation from earlier is not optional. Without it, you will not even notice the shift until it has already cost you.

Price the whole thing before you commit

Add these up and a clear picture emerges. The cheap, fast start you experienced is real, and it is the least representative part of the entire project. Underneath it sits a per-call cost that grows with success, the engineering tax of non-determinism, the ongoing work of evaluation, the maintenance of all the glue around the model, a dependency on a provider you do not control, and a foundation that can change without warning. None of those showed up in the demo. All of them show up in the business.

The point is not to talk yourself out of building on these models. It is to go in with your eyes open, pricing the whole system honestly instead of the one call that made the demo feel free. Model the cost at real volume. Budget the engineering that non-determinism and evaluation demand. Account for the maintenance of everything around the model, and for the risk of standing on ground you do not own.

The founders who get burned saw how easy the beginning was and assumed the rest would match it. The beginning is the cheapest part. Price the rest before you build on it.