Local AI Inference: A Decision Framework for 2026

Local inference pays for itself above roughly 20,000 requests a day. Below that line, hosted open-weight models cost about a quarter as much, and the reasons to own the hardware are control and latency, not savings. I have priced this both ways for small organizations, and the arithmetic has moved less than the marketing suggests.

What changed is the supply side. Open-weight models under permissive licenses now cover the work most small organizations actually have: extraction, classification, summarization, retrieval. Quantization formats such as MXFP4 put a 117-billion-parameter model inside 80 GB of memory. A desktop with 96 GB of unified memory lists at $6,799. None of that makes inference free; it makes the purchase decision real, and it can now be answered with a spreadsheet instead of an opinion.

Four inputs decide it: how sensitive the data is, how many tokens you move, how fast the answer has to come back, and whether you can tolerate a single point of failure. The prices below are list rates from the providers’ own pages at the time of writing.

What a local box actually costs

Two machines bracket the choice for a small organization. Apple’s Mac Studio with the M5 Ultra chip starts at $6,799 with 96 GB of unified memory, configurable to 512 GB, moving data at 1.2 TB/s and drawing at most 480 W. A rented H100 SXM has 80 GB of HBM3 at 3.35 TB/s and a 700 W thermal design point, and Lambda lists it at $3.99 per GPU-hour. A B200 with 180 GB rents for $6.69.

Configuration	Usable memory	Power draw	Cost
Mac Studio, M5 Ultra, 96 GB	96 GB unified	480 W maximum	$6,799 to buy
NVIDIA H100 SXM, 80 GB	80 GB HBM3	700 W TDP	$3.99 per hour

Depreciated over three years, the desktop costs $2,266 a year before power. Renting one H100 for a month costs $2,913, more than a year of the desktop, and the desktop stays yours. That comparison flatters the desktop: the H100 has nearly three times the memory bandwidth and is built to serve hundreds of concurrent requests. Which machine wins depends on how many requests you put through it, which is the part vendors of both kinds leave out.

What the same tokens cost hosted

Hosted open-weight inference has converged on a narrow price band. For gpt-oss-120B, Together AI and Fireworks both charge $0.15 per million input tokens and $0.60 per million output tokens. Fireworks bills batch work at half its serverless rate.

Provider	Model	Input, per 1M tokens	Output, per 1M tokens
Together AI	gpt-oss-120B	$0.15	$0.60
Fireworks	gpt-oss-120B	$0.15	$0.60
Fireworks, batch tier	gpt-oss-120B	$0.075	$0.30

Output tokens cost four times what input tokens cost, and output length dominates both latency and the bill. An organization generating ten million output tokens a day alongside forty million input tokens spends $12 a day at those rates, or $4,380 a year. That number matters more than any per-token comparison, because it is the figure a local deployment has to beat.

The memory arithmetic

Whether a model fits is arithmetic. Weights occupy two bytes per parameter at 16-bit precision, one at 8-bit, and half a byte at 4-bit, so a 70-billion-parameter model needs 140 GB at 16-bit and 35 GB at 4-bit. Databricks measured Llama-2-70B at 140 GB in half precision, and its engineering team notes that 8-bit halves the GPU count while 4-bit lets the same model run on a laptop.

Precision	Bytes per parameter	70B weights	Hardware that holds it
16-bit	2	140 GB	One 180 GB card, or two 80 GB cards sharded
8-bit	1	70 GB	One 80 GB card
4-bit	0.5	35 GB	One 48 GB card or a 96 GB desktop

The KV cache adds a second, variable allocation: batch size times sequence length times head dimension times layer count, doubled for keys and values, times bytes per value, times the number of KV heads. For the same 70B model at 1,024-token context, the cache is 0.31 GiB for a single sequence and 10 GiB at batch 32. Long contexts and high concurrency both come out of the same memory as the weights.

Mixture-of-experts models break the habit of reading parameter count as speed. gpt-oss-120B has 117 billion parameters and activates 5.1 billion per token, and OpenAI ships it natively quantized in MXFP4 so the whole model runs within 80 GB. Capacity is set by the 117 billion; decode speed is set by the 5.1 billion. A 117B model can answer faster than a dense 70B on the same card, which is why “how big is the model” is the wrong first question.

Throughput is set by batching, not by FLOPs

At one request at a time, each generated token requires reading the model’s weights out of memory. Speed is memory bandwidth divided by bytes read per token, and the compute units sit mostly idle. A dense 70B model at 8-bit reads 70 GB per token, so an H100’s 3.35 TB/s caps single-stream decoding near 48 tokens a second before any inefficiency, and a 1.2 TB/s desktop near 17.

Batching changes the denominator. One read of the weights serves every sequence in the batch, so aggregate throughput rises with concurrency while per-user speed falls slowly until the work turns compute-bound. Databricks puts it plainly: for a 70B model, good cost per token only arrives at large batch sizes.

NVIDIA’s product page shows the size of the gap. It cites SemiAnalysis InferenceX measurements of about $0.09 per million tokens at 66 tokens per second per user for gpt-oss-120B served with vLLM on an H100, as of April 2026. At $3.99 an hour, that price implies roughly 12,300 output tokens a second, which is about 190 concurrent streams. The hosted rate comes from filling the hardware, not from cheaper hardware. One tenant on one rented GPU pays dollars per million tokens for the same model.

The break-even calculation

Put both sides on an annual basis. One 96 GB desktop, depreciated over three years, costs $2,266 a year. Power at 480 W around the clock is 4,205 kWh; at the US commercial average of 14.47 cents per kWh in August 2026, that is $608. Two hours a month of an administrator’s time, at the BLS median wage of $47.66 an hour loaded by 1.4 for overhead, adds $1,601. The box costs $4,476 a year to own and run.

At $0.60 per million output tokens and a four-to-one input-to-output ratio, that buys about 20,000 requests a day of 2,000 input and 500 output tokens. Below that volume, hosted wins. Above it, local does. Redundancy moves the line: a second box for failover takes the annual cost to $7,351 and the threshold to roughly 34,000 requests a day. Cloud batch pricing at half rate moves it the other way, to about 41,000.

Local configuration	Annual cost	Break-even against on-demand cloud	Break-even against batch cloud
One 96 GB desktop	$4,476	~20,000 requests a day	~41,000 requests a day
Two desktops, active/passive	$7,351	~34,000 requests a day	~67,000 requests a day

The threshold moves with the details. A model half the size halves the memory requirement, a 240 W box cuts the power line, and an administrator who already runs Linux shaves the staff line. The shape does not move: idle capacity is the cost that decides the answer, and a box bought for peak load spends most of its life at a fraction of it.

At 5,000 requests a day, a plausible size for a 40-person company, local costs $4,476 against $1,095 hosted: four times more for the same tokens, before anyone counts the first month of setup. If local wins at your volume, you are either moving far more tokens than a typical small company or you are paying for a constraint that has nothing to do with price.

What the spreadsheet leaves out

Pinning weights is both the benefit and the burden of owning the box. New model families arrive faster than hardware refreshes, and a larger model can fail to fit the machine you just bought: gpt-oss-120B needed 80 GB, and a successor at 200 billion parameters would need a bigger card. The $6,799 purchase is a two-to-three-year bet on a memory size. Changing models also costs an evaluation you cannot skip, because you cannot tell whether a quantization change or a model swap hurt output without a task-specific eval set. Databricks warns that naive quantization degrades quality and recommends evaluating the inference system rather than the model alone. Every upgrade means a re-run, and the re-run is the part nobody budgets.

A local deployment also makes you the provider. Token accounting per tenant, latency histograms, error rates, GPU saturation, quotas, and a way to answer “why was that answer slow at 2pm” all become your job. Hosted, that work is inside the per-token price.

The capital line is not fixed either. Server DRAM contract prices rose 13 to 18 percent quarter over quarter in the third quarter of 2026, and TrendForce expects quarterly increases through the second half of 2027. A hardware quote has a shelf life measured in weeks, and the memory bill is the one that moves. Staff time is the line most spreadsheets get wrong in both directions: two hours a month is realistic once a deployment is stable, and the first month is closer to forty. The person who spends them usually already has the most to do.

Licenses decide what you may deploy

Open weights are not the same as an open license, and the difference becomes a product decision the moment you ship something that contains a model.

• gpt-oss-120b and 20b ship under Apache 2.0, with no user threshold and no attribution clause beyond the license text.
• Qwen3.5 models are Apache 2.0 as well, which makes them the least encumbered option for embedding in a commercial product.
• Llama 4 carries the Llama 4 Community License: redistribution requires displaying “Built with Llama”, derivative model names must begin with “Llama”, and an organization above 700 million monthly active users needs a separate grant from Meta.

For a company with forty employees, the practical gap between Apache 2.0 and the Llama license is small. Both permit commercial use, and neither asks you to publish weights. The gap opens in two places: when you embed a model in a product you distribute, where attribution and naming turn into interface decisions, and in due diligence, where a buyer’s counsel reads the exact license you accepted two years earlier.

Putting it together: the decision matrix

Seven workload shapes cover most of what a small organization runs. This is the matrix I use to scope an AI deployment before anyone quotes hardware.

Workload	Data sensitivity	Volume	Latency need	Verdict
Clinical notes and patient records	High	Low to medium	Moderate	Local, or hosted under a BAA with zero retention
Contract review on privileged material	High	Low	Batch	Hosted with zero retention and a DPA if counsel approves; local if documents cannot leave
Customer support chat	Medium	High, spiky	Under a second	Cloud
Document pipeline, 20,000 a day, internal data	Medium to high	High, steady	Batch	Local
Month-end analysis burst	Low	Spiky	Hours	Cloud, batch tier
Search over customer records	High	Medium	Under two seconds	Hybrid: index and redact locally, generate in cloud
Air-gapped or offline site	High	Low	Hard real time	Local

Two rows are where most arguments go wrong. Regulated data is usually a contract problem rather than a physics problem. OpenAI’s enterprise terms, updated in January 2026, say API data is not used for training by default and is retained for up to 30 days for abuse monitoring. Zero data retention is available on eligible endpoints, and the company will sign a business associate agreement for HIPAA work. If a supplier will sign those terms, the data argument for owning hardware weakens a lot. Which terms you need depends on the state and sector rules that apply, and the obligation usually follows the data type, not the vendor.

The other mistake is buying for latency. A cloud call adds a network round trip, which matters for a chat interface and almost nothing for a document pipeline. What local hardware actually buys is control of the tail: the response when a provider has a bad afternoon, and the service that keeps running when your uplink does not.

What this means for operators

The decision is which workload carries a constraint that money cannot fix, and whether that constraint is worth $4,476 a year. For most organizations, the answer is one workload rather than a platform: the pipeline that touches regulated records, or the site with a bad connection. Everything else belongs on a hosted API, where someone else carries the utilization risk.

The cost argument cuts against local at small volume and for it at large volume, and the crossover sits far higher than the sellers of desktop AI hardware imply. A single workstation is also a single point of failure, which is why the two-box number matters more than the one-box number that appears in every review. My view: at 5,000 requests a day, buying a box to save money is a mistake, and buying one because a contract forbids sending the documents out is a different decision that happens to be defensible. The same test applies to AI adoption in general: the workflow decides, and the model is the last variable.

There is a security dimension the cost table hides. A local inference server is another internet-adjacent system to patch, monitor, and segment, and small organizations are already behind on that work. A GPU box with a web interface and no named owner makes the estate worse. The controls in the tier-one security program come before the accelerator, and an agent with credentials is a larger risk than the same agent on a hosted API.

What to do

1. Count the workload first. Pull 30 days of request volume, input and output token counts, peak concurrency, and the latency your users actually need. Most local-versus-cloud arguments collapse once the volume is on paper.
2. Run the matrix on one workload. Choose the single candidate with the strongest constraint, price it both ways, and leave the rest of the estate alone. A pilot that moves everything at once cannot tell you which decision was right.
3. Price redundancy into the capital request. Two boxes, or one box plus a documented cloud fallback with a tested cutover. The one-box number is a lab result, not a production plan.
4. Test the model you will actually ship, at the quantization you will actually run. Build a 100-example eval set from your own documents and compare 4-bit against 8-bit before you buy memory to hold it.
5. Negotiate the cloud side before you rule it out. Ask for the batch tier, zero data retention, a data processing addendum, and a business associate agreement if health data is involved. Batch pricing alone doubles the volume at which local breaks even.
6. Name the owner and the runbook. One person accountable for patching, upgrades, and capacity, with an alert path to a human and a patching cadence that matches the rest of your estate.
7. Set a review date with a kill criterion. If sustained utilization stays under a third of capacity for two quarters, consolidate to the hosted API and redeploy the capital.

This quarter, pick one workload and put two numbers on it: requests per day, and cost per million output tokens under each option. If the answer lands under 20,000 requests a day, keep it hosted, spend the difference on identity, patching, and backups, and revisit the question when volume triples or a contract forces it.

Sources

1. Pricing — Together AI, accessed 2026-11-02.
2. Serverless Pricing — Fireworks AI, accessed 2026-11-02.
3. GPU Cloud Pricing — Lambda, accessed 2026-11-02.
4. Buy Mac Studio — Apple, accessed 2026-11-02.
5. Mac Studio Technical Specifications — Apple, accessed 2026-11-02.
6. NVIDIA H100 GPU — NVIDIA, accessed 2026-11-02.
7. LLM Inference Performance Engineering: Best Practices — Databricks, 2023-10-12 (updated 2026-09-30).
8. Introducing gpt-oss — OpenAI, 2025-08-05.
9. Enterprise privacy at OpenAI — OpenAI, 2026-01-08.
10. Llama 4 Community License Agreement — Meta, 2025-04-05.
11. Qwen3.5-9B model card — Hugging Face, accessed 2026-11-02.
12. Server DRAM Contract Prices Expected to Rise 13-18% QoQ in 3Q26 — TrendForce, 2026-07-09.
13. Electric Power Monthly, Table 5.6.A — US Energy Information Administration, 2026-10-23.
14. Occupational Outlook Handbook: Network and Computer Systems Administrators — US Bureau of Labor Statistics, 2026-08-27.