Services

Page

Services

Page

Cloud AI API vs Self-Hosted LLM: Cost and Control Trade-offs

Diana Zander Research Muse

12 min30 Oct 2026

Want to discuss
your project?

The right choice is the one your data rules and your traffic can support. Four paths, four questions and a decision in five lines.

Key takeaways

Why it matters: the choice sets your cost curve, your data exposure and your team's workload for years.
What to do first: write down your monthly token volume and your data rules before you compare any prices.
Cloud APIs win at low and uneven volume, because you pay only for what you use.
Self-hosting pays back at steady high volume or under strict data rules, and only once a team can run it.
Many teams end with a mix, so plan the switch before you need it.

Most teams ask "API or our own server?" as a price question. Prices decide less than three other things: how much traffic you will really send, where your data is allowed to go, and who will be on call when the model server stops answering.

Step 1: four questions that decide it

Answer these before you open a pricing page. Each one removes at least one path.

How many tokens a month, and how even is the traffic? A steady stream suits a server you pay for by the hour. Sharp peaks suit a service you pay for by the token.
Which data will the model see? Public text and your own marketing copy are one case. Medical, financial or personal records are another, and some rules decide where that data may travel.
How fast must an answer arrive? A chat reply tolerates a second or two. A voice assistant or a checkout step does not. Distance to the model and queueing both count.
Who will run it? Self-hosting needs people for deployment, monitoring, updates and security. If nobody owns that work, the cheapest server becomes the most expensive choice.

Option 1: Cloud API. The pay-as-you-go path

You send a request to a provider and pay per token. There is no hardware to buy and no model to deploy. You get the provider's newest models quickly, and you scale from ten requests a day to ten thousand without a plan.

The trade is control. The provider sets prices, rate limits, model versions and retirement dates. Your data leaves your network, so contract terms about retention and training use matter as much as the price list.

Pick it when: volume is low or uneven, you need the strongest models today, or you want to ship in weeks.
Watch for: price changes, rate limits, a model being retired on a date you did not choose, and data terms that your compliance team has not read.

Option 2: Self-hosted open-weight model. The fixed-cost path

You rent or buy GPUs, run an open-weight model on your own serving software and pay for the machine whether it is busy or idle. Cost per token falls as utilization rises, so steady high volume is where this path earns its keep.

You also gain control: data stays inside your network, versions stay put, and you can tune the stack for your latency target. You also take on the work of running it.

Pick it when: traffic is steady and high, data cannot leave your environment, or you need a fixed price and a fixed model version.
Watch for: idle GPUs, the licence on the model you choose, and the engineer time that every comparison table leaves out.

Option 3: Private endpoint from a cloud provider. The middle path

Major cloud platforms offer hosted models in your own tenant or region, with network isolation and enterprise terms. You keep the pay-per-use model and the managed operations, and you gain stronger answers to questions about where data goes.

Pick it when: you need regional hosting or private networking but cannot staff a model platform.
Watch for: a narrower model list, extra fees for isolation, and quotas you must request in advance.

Option 4: Hybrid. Route each task to the right place

Most mature teams end here. A router sends simple, high-volume, sensitive or latency-critical work to a small self-hosted model, and sends hard reasoning tasks to a frontier API.

Pick it when: your workloads differ a lot in difficulty, sensitivity or volume.
Watch for: the cost of a router, two sets of prompts to test, and a second system to monitor.

A worked break-even example

The arithmetic below is illustrative. Replace every number with your own before you decide.

Several public guides put a rented H100 GPU at roughly $2.89 to $4.29 an hour, with a median near $3.81. Running one around the clock for a 730-hour month costs about $2,781 before any staff time.

That monthly cost buys a fixed amount of capacity. The break-even volume is that cost divided by what the API would charge for the same tokens.

Three corrections move the break-even further out:

Utilization. A GPU that is busy 60% to 70% of the time costs about a third more per useful token than one that is always busy, and quiet nights and weekends push it lower still.
People. Deployment, monitoring, security updates and on-call time are real cost. One part-time engineer can exceed the GPU bill at small scale.
Comparison point. Break-even against a frontier API arrives much sooner than against the same open model on a cheap hosted endpoint. Compare with the API you would actually replace.

The quiet risk: costs that appear after launch

The bill you plan for is tokens or GPU hours. The bill you get includes more:

Retries and long prompts. Context windows that grow with every product feature multiply token counts without anyone deciding to.
Evaluation and testing. Every model change needs regression tests, and every test run costs tokens or GPU time.
Egress and storage. Logs, embeddings and cached prompts live somewhere, and that place sends invoices.
Migration. Moving from one path to another takes prompt rewrites, new evaluations and a parallel run. Plan the exit when you plan the entry.

Seven mistakes we see most often

Choosing on list price alone. Price per million tokens hides people, idle time and evaluation.
Self-hosting for a spike. A server sized for the peak sits idle the rest of the month.
Skipping the data terms. Retention and training clauses are easy to read once and expensive to discover later.
Ignoring the licence. An open-weight model can still carry user thresholds and use restrictions.
No evaluation set. Without your own test questions, you cannot compare two models or two paths.
One path for every task. Easy and hard requests do not need the same model.
No exit plan. Prompts written for one provider rarely run unchanged on another.

Questions to ask a vendor or an internal team

Which model versions will be available a year from now, and how much notice do we get before a retirement?
Is our data retained, for how long, and is it used to train anything?
Where is the data processed, and can we pin a region?
What are the rate limits at our expected peak?
How is the serving stack monitored, and who is on call?
What does a model upgrade cost in testing time?
How do we move to another path if prices or terms change?

Terms in plain words

Token: a small piece of text, roughly three quarters of a word, that models read and write. Providers price by the token.
Open-weight model: a model whose trained weights you can download and run, under a licence.
Inference: running a trained model to get an answer.
Utilization: the share of time a GPU is doing useful work.
Latency: the time between a request and the answer.
Private endpoint: a hosted model reachable only from your own network.

In Lazy Ants

We treat this as a sequence of decisions, in the order above:

We measure your real volume, peaks and latency targets before we compare prices.
We map which data the model sees and which rules apply to it.
We build a break-even model from your numbers, not from a vendor's.
We run an evaluation on your own questions across two or three candidate setups.
We design the routing and the exit plan, so the choice stays reversible.

If you want an outside read before you commit, our AI automation services start with a review of your use case and constraints. For adjacent decisions, see what an AI chatbot costs and when fine-tuning pays off.

Step 2: decide in five lines

If your data must stay inside your environment, plan for Option 2, or Option 3 if a private endpoint meets your rules.
If volume is low or uneven, start with Option 1.
If volume is steady and high, price Option 2 with a realistic utilization and a staff estimate.
If your workloads differ a lot in difficulty, plan Option 4 from the start.
Whatever you choose, write the exit plan and keep one evaluation set you can rerun.

FAQ

Is self-hosting cheaper than an API?

Only at steady high volume, and only when utilization and staff time are counted. At low or uneven volume the API is usually cheaper.

╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

How many tokens a month do I need to make self-hosting worthwhile?

There is no single number. Published break-even points run from hundreds of millions to several billions of tokens a month, depending on which API you replace and how busy the GPU stays. Compute it from your own throughput, price and utilization.

╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

Can I switch later?

Yes, if you plan for it. Keep prompts and tests independent of one provider, and keep an evaluation set so you can compare a new path in days.

╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

Are open-weight models free for commercial use?

Not always. Licences differ by model family and by checkpoint, and some carry user thresholds or use restrictions. Read the licence file for the exact model you deploy.

╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

What if our data is sensitive?

Start from the rules that apply to it, then choose among the paths that meet them. A private endpoint or a self-hosted model is often the answer, but the contract terms of a cloud API can also qualify. Ask the vendor in writing.

╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

Do we need our own GPUs to self-host?

No. Most teams rent GPUs by the hour. Buying hardware only makes sense with long, predictable demand.

╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

What is the safest first step?

Run your real prompts against an API with an evaluation set, record tokens and latency for two weeks, and use those numbers for every later decision.

Sources: public cost comparisons of self-hosted and API inference from engineering blogs and vendor write-ups, including Cohorte, Cloudzy and DEV Community; secondary summaries of open-weight licence terms. All figures are estimates and prices change, so check current pricing, licence files and contract terms before you plan.