When Local Actually Beats the API

The naive comparison, and why it misleads

Local inference has no per-token price, so it looks free. It is not. You have just moved the cost from a variable to a fixed one, and added two costs that do not appear on any invoice.

The real comparison:

	Hosted API	Local
Marginal cost per call	$0.02–0.10	≈$0
Up-front cost	$0	$1,500–4,000
Throughput	50–150 tok/s	8–50 tok/s
Capability	Frontier	Meaningfully behind on agentic work
Data leaves the machine	Yes	No

The break-even calculation

months to break even = hardware cost / monthly API spend

Against a typical solo-dev workload:

Monthly API spend	$2,400 machine pays for itself in
$28 (fast tier)	86 months
$56 (balanced tier)	43 months
$140 (frontier tier)	17 months
$700 (heavy team use)	3.4 months

For one developer on a balanced-tier model, hardware never pays for itself before it is obsolete. That is the honest answer, and it is why "local to save money" is usually wrong.

Run your own numbers in the calculator.

When local genuinely wins

Cost is rarely the reason. These are:

1. The data cannot leave. Regulated environments, client code under NDA, personal material. No amount of API pricing changes this. It is a hard constraint, and local is the only answer.

2. You are already offline. Flights, poor connectivity, air-gapped networks.

3. Volume is enormous and the task is simple. Classifying ten million documents with an 8B model. Here the arithmetic does flip, decisively.

4. You want to learn how the stack works. A real reason. Just do not file it under cost savings.

5. You already own the GPU. If the hardware is sunk cost from gaming or ML work, the marginal cost really is close to zero and the break-even question disappears.

The capability gap is the real cost

A 30B-class model at Q4 handles bounded, well-specified edits well. It does not handle long-horizon agentic work, multi-file refactors, ambiguous debugging, planning across a session, anywhere near a hosted frontier model.

If a local model needs three attempts where a hosted one needs one, and your time is worth anything, the free tokens were expensive.

Sizing the machine

VRAM is the binding constraint. Rough arithmetic:

VRAM ≈ (parameters × bits per weight / 8) + KV cache headroom

Class	Q4 VRAM	Runs on	Throughput
8B	~6 GB	Any 8 GB GPU	40–80 tok/s
30B	~20 GB	RTX 4090, 36 GB Apple Silicon	25–50 tok/s
70B	~42 GB	2× 24 GB, 64 GB Apple Silicon	8–20 tok/s