How far can a $2,000 homelab rig get toward frontier inference?

A 125-billion-parameter model runs at 77 tokens/second for $3,250. That is faster than the same model on the card the engine's own documentation benchmarks it against, and flat across every context depth tested. Frontier inference is still out of reach, and this paper says why in arithmetic rather than in adjectives.

Evidence marks: [M] measured here · [D] derived from measurement · [S] sourced · [T] theory. Full data, harnesses and failures are published alongside this summary.

The answer

Throughput is bandwidth divided by the bytes moved, times the tokens produced per pass:

tok/s  =  (bandwidth ÷ bytes-per-token)  ×  tokens-per-pass

Measured residual: 1.0000× across three queue depths [M]. Not a fitted curve — the bound. Every optimisation below is one of those three terms.

$3,250 delivers 125B-class inference at 1.6× the decode of a dense 27B on the same hardware — while running a model 4.6× larger.

Where the wall is

Not PCIe width. The ×8 Gen4 link was worth 8 % on prefill and 2 % on decode against a ×1 Gen3 link [M]. A bandwidth-starved rig would have shown far more.

Not card count — card sequentiality. Three cards in a layer split deliver roughly one card's bandwidth, because layer N+1 cannot start before layer N finishes. The engine reports it directly: graph splits = N+1.

Three cards against one, VRAM-resident:

Adding cards did not raise the ceiling; it delayed the fall. That is a capacity wall, not a bandwidth wall — which is why the fix was memory, not another GPU.

Frontier models remain impossible, and the arithmetic is short: a 400B dense model at 4-bit needs ~200 GB; a frontier dense model ~700 GB–1 TB. This rig has 48 GB of VRAM and 128 GB of RAM, on PCIe with no P2P. No tuning addresses that.

The measurements

The file tier was the floor

At 32 GB the engine cannot hold the experts, so it reads them from NVMe every token: 1,145 MB per request, 2,492 major faults, 13.3 % of expert traffic from storage [M]. Decode 23.8–39.8 tok/s, and the engine died after two to four requests.

The second tier removed it

At 128 GB, with --resident-experts, the engine page-locks the experts in RAM. In its own words [M]:

resident RAM mode: 28.61 GiB of experts in RAM (page-locked) ... (no file reads)
expert tiers:      RAM blobs; files 0 blobs

files 0 blobs. The storage path is gone.

Mean 77.1 tok/s; 10 of 10 sequential requests succeeded.

Two things were not predicted. Prefill rate rises with depth — short prompts are dominated by fixed per-request overhead, long ones amortise it. And decode is flat across a 60× range of prompt depth, where the file-tier configuration collapsed. A single depth would have reported whichever effect dominated that prompt, which is the argument for measuring a curve rather than a point.

Against other people's numbers

At short context this rig matches the vendor's card. At 128 K it is 25 % ahead — because depth costs almost nothing once the experts are resident, which is exactly what the second tier bought.

The community one-card figure lands inside this rig's file-tier band. Same configuration, same task, different storage — and the agreement is the strongest independent check here.

The honest caveat: conditions differ between every row. This compares configurations a person could own, not silicon on equal footing.

What it cost

The $1,250 changed three things at once — RAM, CPU, and an unplanned PCIe Gen3→Gen4 link from the CPU swap. So "it bought 1.8×–3.4×" is a combined claim; separating them would need a controlled build, which has not been done.

The gain was predicted before the hardware arrived. At 32 GB, 13.3 % of expert traffic came from storage. Only that fraction can be removed by RAM, which bounded the gain at 1.0×–2.9× [D] — predicting 40–90 tok/s. The measurement was 70.3–80.9. Stating the projection in advance is why it can be checked.

One lever is free and easy to overstate: XMP took memory from 2133 to 3200 MT/s for +50 % where the working set did not fit in VRAM — and for +0.4 %, inside the noise, once the model was resident [M]. Both measurements are here because publishing only the first would imply a lever that does not exist.

What is not established

• Why three cards ceiling at 47.4 tok/s. It holds across every depth and concurrency level tested, which points at a bandwidth wall — but whether the cards are read sequentially or saturating something shared is untested.
• Card count and offload are confounded in every 1-vs-3 comparison: the "one card" arm was a partial-offload configuration, not a clean baseline.
• Three cards with resident experts was not measured. The serialisation above suggests it may not help.
• The seven-quantisation sweep is prepared and running, not yet in these numbers.

Two directions forward

Neither is a recommendation, and neither is dismissed.

Finish the levers already evidenced. Every lever above has a measured effect on a real configuration; the work is execution. For: measured effects, known costs, low risk. Against: the largest remaining lever is capacity, and capacity is the expensive one — this improves a working configuration without changing what is possible.

Abandoned software on discarded hardware. Professional accelerators are discarded while functional, and the software that would use them well is often unmaintained. For: the hardware is cheap, and the bandwidth-per-dollar metric that disfavours it is specifically a decode metric — the workload consumer cards already win. Against: that disfavour is measured, not assumed, and it constrains which workload to attempt.

The criterion: a direction is worth pursuing when its claim can be made false by one measurement. Both pass. Neither has been attempted.

Method, and why it is in the summary

This project produced measurements that were wrong in ways that looked plausible, and the three that shaped the method are worth naming:

• 89 tok/s prefill — from a 34-token prompt. The real rate is ~700. An 8× error from an unstated denominator.
• 1,178 tok/s decode — from a two-point timing method that sent the same prompt twice; the second call hit the prefix cache and came back faster than the first.
• Nine failed experiments on one fault — every configuration failed identically until the binary was questioned rather than the configuration. It was a CUDA 12.8 build on a Blackwell card whose installer wanted CUDA 13. Same source, same pack, same card, CUDA 13: every failing prompt passed.

The rules that came out of it: one variable per arm · state the denominator · separate cold from warm · trust the engine's own telemetry over your theory · record the binary's version, not the one you meant to install · publish the failures.

The full paper — methods, bug catalogue, retractions, artifacts, reproducible harnesses — is published alongside this summary, with every figure traceable to the named script that produced it.