Strata on 3× RTX 5060 Ti: precision doesn't predict speed. I measured five quants and found out why.

A follow-up to my 125B-on-consumer-hardware study. The short version: the equation from that post still holds. I also found the places where it doesn't.

So my last post ended with a law:

tok/s = (bandwidth ÷ bytes-per-token) × tokens-per-pass

Residual 1.0000× across three queue depths [M]. I called it the bound, not a fit.

This post is what happened when I tried to break it by changing the quantisation... It broke... by up to 41% ... The reason it broke is the most useful thing I've measured.

Five Quants, Same Rig, Same Method

Same 125B model, 131K window, cold-prompt discipline. Every number comes from a prompt the cache had never seen , so none of these are a cache hit:

Look at rows 2 and 3.

Q2_0 at 2.19 bits is slower than IQ3_XXS at 3.06 bits.

The kicker is row 2 was reading its experts off a SATA (spinning disk) BEATING row 4, which pins 47 GiB in RAM.

That is not a small anomaly. It means bits per weight do not order these results at all.

Engine's Log (Mechanism Responsible)

For IQ3_XXS:

expert cache 4748 slots, 7.70 GiB of VRAM R4 hit path ON - resident experts are computed on the GPU STRATA_ROUTE_TAIL_SKIP=7 ON: a missed expert routed at rank 7 or lower is skipped

So: a VRAM expert cache holding 7.7 GiB, the GPU computing resident experts itself instead of streaming them back, and rank-7 misses being skipped outright.

Bits per weight set the size of the problem. The path those bits travel sets the speed. A file-backed model with that machinery outruns a fully resident one that lacks it.

This is why the law held so cleanly in the first study and why I couldn't see the boundary from there: I had only ever varied depth and quantity of data. I had never varied the path. Vary all three, and the third one dominates.

I predicted all five before measuring

One quant lands within 4%. One misses by 41%.

A Law With Scope

What tok/s = (bandwidth ÷ bytes-per-token) × tokens-per-pass actually says is; a fixed bandwidth must move a fixed number of bytes.

It says nothing about how many of those bytes ever get fetched. That's the mechanism, and the mechanism is not in the equation.

The model's native window is 262K. Its reachable window is 1M, via YaRN ×4 [S].

I predicted 1M was impossible.

My reasoning: a dense-attention model at that depth needs ~70GB of KV, which doesn't fit alongside the weights. I put that reasoning in writing before checking.

Measured: pinned KV at 1M is ~12.4 GiB [S]. I was wrong by six times, because this model keeps constant state in state-space layers, accumulates KV only on intermittent full-attention layers, and indexes sparsely. Dense-attention assumptions don't transfer.

So I measured it. Cold prompt, 1M window: 807,866 tokens, entirely cold prefill 851 tok/s decode 28.4 tok/s engine loaded in 1.5 minutes, 53 GB of RAM still free

807,866 tokens — the exact number, not "1M," because the exact number is the one that survives someone checking.

And one honest drop: decode does fall past the trained boundary — 28.4 against the 43.3 the same model does at 262K. Flat decode was a property of a resident model working inside its window. It is not a law that survives outside it.

Two edges I hit, so you don't have to

IQ1_M does not exist for this model. I predicted 90.8 tok/s for it. The family ships Q2_0, IQ2_XS, IQ3_XXS, IQ3_S — IQ1_M is Coder-only. My prediction was for a model that isn't there.

UD-Q4_K_XL is not measurable on this box. 111GB of model and 77GB of experts against 88GB free NVMe. It could only run disk-bound off a platter, which measures the disk, not the quantisation. Published as a gap rather than an artifact.

What I still can't separate

These five points vary in bits-per-weight and delivery path simultaneously. So the mechanism story above is inferred, not isolated — and I want to be careful about that, because "the path matters" is exactly the kind of claim that sounds proven when it's only correlated.

The clean experiment is one quant served two ways: same weights, resident versus file-backed. IQ3_XXS already has an mmap pack. Building it a resident pack is the next measurement, and it would turn this table from a correlation into a controlled comparison.

Until then, this post claims the ordering is not explained by bit count. It does not yet claim to have proven why.

What this cost to learn

Three failures worth the words:

A 94GB model configured onto a spinning disk. Setup chose the path by RAM budget, NOT by device speed. With 55 of 60GB cached, the ~8% of expert reads that miss were random platter access. Moved it to NVMe — 13.9 minutes, byte-count verified. Rule: check rotational on the model's filesystem before you measure a file-backed model.

The through-line

My first post said: the wall is arithmetic, and here is the arithmetic.

That's still true. What I now know is that there are two arithmetic walls, and the second one is about movement rather than size.

• Size: can the working set fit? (2-bit does, 4-bit barely doesn't at 125B in 128GB)

• Movement: once it fits, how do the bytes actually get to the ALUs?

The first post measured the first wall. This one measured the second, and it's the bigger lever, because it's where a 3.06-bit model on a spinning disk can beat a 3.44-bit model pinned in RAM.

[M] measured on this rig · [D] derived · [S] external source

Rig unchanged: Ryzen 9 5900XT, 128GB DDR4-3600, 3× RTX 5060 Ti 16GB. Five quants, all cold, all published. Prior study linked at the top.