BoteamLocal LLM vs cloud API: agent architecture choices
By Bob · 2026-08-16 · 3 min read
A 3-axis framework: data, unit cost, and latency, to pick between running models locally or calling a cloud API. Built for a team of one.
You ship an AI feature. The first question is not "which model". It is "where does the model run".
Local or cloud. That choice quietly decides your cost, your privacy, and how fast the feature feels.
Most solo founders pick cloud by default because it is the path of least resistance. That is fine. But default is not a decision. This note gives you a way to decide on purpose.
The framework is three axes:
- Data: what leaves the machine.
- Unit cost: what one call costs at your volume.
- Latency: what the user feels.
Score each one quietly. Then pick. No hype here, just the trade.
Axis 1: Data, what leaves the machine
Cloud API means your prompt, your user's input, and the output travel to a third party. For most B2B SaaS that is acceptable. For some it is not.
Ask two questions:
- Does the input contain anything you would not put in an email to a stranger?
- Does your user's contract, or your industry, restrict where data can be processed?
If the answer to either is yes, local moves up the list.
Local means the model runs on your hardware or a machine you control. Nothing leaves the box. That is a clean answer to privacy questions, and it is a real marketing line for privacy-sensitive buyers.
The cost is yours to carry: you buy the hardware, the electricity, and the setup time. For a solo founder, setup time is often the real cost, not the hardware.
Axis 2: Unit cost, what one call costs at your volume
Cloud pricing is per token. Local is a fixed cost you already paid.
The crossover depends on volume. Rough shape:
- Low volume (hundreds of calls a day): cloud usually wins on cost and effort.
- High volume (thousands of calls a day, sustained): local can beat the per-call bill if the calls are repetitive.
- Bursty volume (peaks and silence): cloud wins. You pay only when you run.
The mistake is comparing a $0.01 API call against the price of a GPU. That is not the comparison. The comparison is your total monthly cloud bill against the amortized cost of running local: hardware, power, and your time to keep it running.
The local tooling has matured enough that this is now a real option. The GitHub local-llm topic alone holds 4,000+ public repositories, and single-file servers that speak the OpenAI API (drop-in base_url swap) have been stable for years. You can move a cloud app to local without rewriting the code.
Axis 3: Latency, what the user feels
Cloud adds network round trips. Local removes them.
For a chat surface, a few hundred milliseconds rarely matters. For a tool that runs in a loop, an agent that calls a model many times in a row, latency compounds fast. Ten sequential calls at 400ms each means four seconds of waiting. Run those locally at 40ms each and the same loop is under half a second.
This is the quiet reason multi-step agent workflows feel slow on cloud. It is not the model. It is the accumulated round trips.
If your feature is an agent that makes many calls, measure the total loop time, not one call. That number decides the axis.
The decision table
| If your feature... | Start with |
|---|---|
| Handles sensitive data, needs a privacy line | Local |
| Runs many sequential calls in a loop | Local |
| Has steady, high, repetitive volume | Local, after a cost pass |
| Is low volume, bursty, or still being validated | Cloud |
| Needs the newest model quality immediately | Cloud |
| Is a v1 you have not proven yet | Cloud, until numbers say otherwise |
The last row is the one most solo founders should obey. Prove the feature on cloud. Move to local only when the numbers, cost or latency, tell you the switch pays for itself.
The decision checklist
Run this in under ten minutes:
- Write down what calls your feature makes per day. Rough numbers are fine.
- Ask the two data questions. If either is yes, local moves up the list.
- Estimate your monthly cloud bill for this feature at today's volume.
- Unless a data or volume trigger fires, keep cloud for now. Prove the feature first.
- Write one trigger line you will act on: cost, latency, or a data question from a customer.
- Re-run this checklist only when a trigger fires. That is the whole system.
No dashboard, no forecast, no guess. Set the triggers, then move on to the work that matters.