Requantizing Qwen3.8-Flash-Next for my own machine

16 November 2026
Q4_K_XL (the first build)
Base modelQwen3.8-Flash-Next
File102.5 GB, 4 shards
Context262,144 tokens
Quality
Perplexity2.9165 ± 0.0363
KLD vs UD-Q4_K_XL0.0282
Down error vs BF160.00667
Speed
Prefill, first 33k128 tok/s
Prefill at 262k57 tok/s
Decode at 262k14.4 tok/s
Setup
MachineAMD Strix Halo
Ryzen AI MAX+ 395
GPURadeon 8060S
Memory128 GB DDR5, 8000 MT/s
SystemWindows 11, Vulkan
Enginellama.cpp b10715

I know Qwen4 is just around the corner, but I decided to give my Qwen3.8-Flash-Next GGUF another go. It's quantized specifically for my machine: 128 GB of DDR5 at 8000 MT/s on an AMD Strix Halo (Ryzen AI MAX+ 395, Radeon 8060S), running Windows 11.

With the release of llama.cpp 1.0.0 I reran my personal benchmarks and noticed there were gains to be had. I wanted them, because I'd sometimes run out of memory with my current GGUF: the model would complain that it couldn't run a build or a test for lack of RAM. So I went for it.

Naturally, everything was built on my daily driver, running my everyday Windows, so the speed and performance you see here come from the actual machine, not from some impossible sandboxed environment. The card on the right is where I started.

And then, of course, the journey begins.

Where the memory goes

A 128 GB Strix Halo on Windows can give its GPU memory in one of two ways: a fixed carve-out of 96 GB, or dynamic allocation with the shared-memory slider at its 124 GB maximum, where one process can put about 80 GiB on the GPU. This model's 28.8 GB n-gram table never leaves system RAM, so only the second mode works.[1]

llama-server -m Q4_K_XL-IQ4NL/Qwen3.8-Flash-Next-Q4_K_XL-IQ4NL-00001-of-00004.gguf \
  -c 262144 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -ub 512 -np 1 \
  -lm mmap --lazy-mode on --cache-ram 0 --ctx-checkpoints 8 --jinja \
  -md mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf --spec-type draft-mtp,ngram-mod
q8_0 keys and values
0.012
q4_0 keys, q8_0 values
0.019
q4_0 keys and values
0.026
What each KV-cache setting costs, as KLD against an f16 cache at 2048 tokens. None changes decode speed.

Speed across the context

050100150200250064k128k192k256kcontext (tokens)prefill (tok/s)unsloth b10715b11160-mix, int8 onb11160-mix, int8 off (mine)
  • unsloth b10715
  • b11160-mix, int8 on
  • b11160-mix, int8 off (mine)
Prefill speed as one conversation grows to 262k tokens, by engine. The int8 matmul reads the prompt faster, but changes what the model sees; I run it off.

The builds compared

UD-Q4_K_XL, 111.3 GB
2.9167 ± 0.0365
Q4_K_XL-IQ4NL, 102.5 GB
2.9092 ± 0.0362
UD-IQ4_XS, 93.7 GB
2.998 ± 0.038
2.852.902.953.003.05
perplexity on wikitext-2, lower is better
Perplexity with its error bars. The new build is level with the 111 GB reference and clearly ahead of the next size down.

In daily use, with gatto

under 25k
28.4 tok/s
25–50k
20.1 tok/s
50–100k
17.0 tok/s
at 262k
15.8 tok/s
Decode speed in daily use with gatto, by context length (medians).

Old and new, side by side

Q4_K_XL-IQ4NL (the new build)
Base modelQwen3.8-Flash-Next
File102.5 GB, 4 shards
Context262,144 tokens
On the GPU78.3–78.6 GiB
Quality
Perplexity2.9092 ± 0.0362
KLD vs UD-Q4_K_XL0.0304
Same top token94.40 %
Down error vs BF160.00490
Speed
Prefill, first 33k159 tok/s
Prefill at 262k67 tok/s
Decode at 262k15.8 tok/s
Decode in daily use28.4 / 20.1 / 17.0 tok/s
under 25k / 25–50k / 50–100k
Get it
Hugging Facebitlamas/Qwen3.8-…
First buildNew buildChange
File size102.5 GB102.5 GBsame
Squared error vs BF160.006890.00583▼ −15.4 %
Weighted error vs BF160.006670.00490▼ −26.5 %
Perplexity2.91652.9092▼ −0.25 %
Prefill, first 33k128 tok/s159 tok/s▲ +24 %
Decode at 262k14.4 tok/s15.8 tok/s▲ +10 %

Conclusion

The new build is on Hugging Face as bitlamas/Qwen3.8-Flash-Next-GGUF, with the raw logs for every number.[2]

References

  1. ^ Qwen3.8-Flash-Next GGUF, model card, “Why this size, on Windows”.
  2. ^ logs/, the raw measurements.
Retrieved from https://gatti.sh/log/requantizing-qwen3-8-flash-next