I Built a Custom Blackwell Kernel for Qwen3.8-27B

When I decided that we should try to run Qwen3.8-27B at more than 200 tokens per second on a single RTX PRO 6000, the point was not simply to produce a large number. We were already above 100. I wanted to know how much performance was still hidden below the level where most inference frameworks stop giving you control.

This is directly related to why I am bullish on open Chinese models. Alibaba can release the weights, but the more important fact is that we are then allowed to inspect and optimise the entire system around them. Open weights do not make inference fast by themselves. They make it possible for us to do the work.

What the custom path actually does

Qwen3.8-27B is not a conventional 64-layer Transformer. Forty-eight of its layers use Gated DeltaNet, which carries a recurrent matrix and a four-tap convolution history through the sequence. Only sixteen layers use full attention. This matters because speculative decoding cannot simply append some draft tokens to a KV cache: it must also decide what happens to the recurrent state of those forty-eight GDN layers when only part of a proposed block is accepted.

The quantisation is also more specific than “FP4”. In layers 0–55, the large MLP gate, up and down matrices retain the source checkpoint's packed NVFP4 E2M1 values and block scales. Attention and GDN projections, the output head and the final eight MLPs remain row-scaled FP8. Norms and convolutions remain BF16, while the GDN decay controls and recurrent matrices remain FP32. The runtime therefore has to execute one deliberately mixed numerical graph rather than treating every matrix as a four-bit weight.

I used NInfer as the foundation because it already exposes that graph as a closed Qwen target rather than reconstructing it at runtime from a Transformers model class. The converter preserves the NVFP4 and FP8 code words, then physically joins matrices only where they share the same input and rounding boundary. The resulting .ninfer image contains 112 fused NVFP4 parents and 146 FP8 parents, with the input and weight divisors required by each NVFP4 site bound into the artefact. There is no GGUF conversion and no second quantisation pass.

At execution time, NInfer uses separate routes for the shapes which actually occur. Small decode rows use hand-written activation-16 NVFP4 kernels; larger prefill tiles can use W4A4 tensor-core kernels after quantising the activation into Blackwell's FP4 format. Q/K/V/Z projections, residual additions and MLP stages have fused leaves so the intermediate activation is not repeatedly written to and reread from HBM. The complete single-request decode schedule, workspace addresses and kernel topology are captured in an exact-batch CUDA Graph at startup.

The DFlash2 part is equally model-specific. Five small draft layers consume target features from layers 5, 19, 33, 47 and 61. They propose seven new tokens as one eight-position block using a 2,048-token noncausal local window, grouped two-tap dynamic convolutions and the full 248,320-row target language head. At every draft step the runtime keeps a stable top sixteen, then applies the trained rank-256 predecessor/successor selector to construct the next token in the chain.

The target model verifies that block in one pass. During verification, NInfer records the represented convolution, key, value and gate inputs for every candidate GDN transition without publishing them as the new model state. Once the verifier knows the accepted prefix, a ReplaySSM fold applies only those accepted transitions to the persistent GDN matrices and convolution history. Rejected candidates never become visible state. Full-attention KV follows the same commit boundary.

That is the custom path: not one isolated matrix-multiply kernel, but a fixed Qwen3.8 execution schedule joining native low-precision operators, DFlash2 proposal, target verification and transactional recurrent-state commit.

How this differs from llama.cpp and SGLang

The llama.cpp route is designed around portability. Our incumbent loads a Q5_K_XL GGUF target and a separate Q4 MTP GGUF through the general GGML graph, then proposes up to four draft tokens. It is an excellent deployment format, and its broad hardware support is precisely why we use it. It does not preserve this checkpoint's native mixed NVFP4/FP8 layout, execute the five-layer DFlash2 companion or use the model-specific GDN replay schedule described above.

SGLang is a closer comparison. Its evaluation arm loads an NVFP4 Safetensors checkpoint, uses FlashInfer attention, keeps the Mamba/GDN state in BF16 and can attach the same DFlash2 checkpoint through its general speculative-decoding interface. That makes it the right semantic reference. It still passes the model through a general Python/PyTorch executor, scheduler, cache manager and backend-dispatch system designed to support many architectures and concurrency patterns.

NInfer deliberately gives those things up. It supports a small registry of exact artefact identities, one resident model and a startup-fixed request capacity. In return, the converter, memory plan, operator selection, speculative state transaction and CUDA Graph are all written for this Qwen topology.

Path	Weight representation	Draft path	Runtime boundary
llama.cpp	Q5_K_XL GGUF target	Q4 GGUF MTP, up to four proposals	Portable GGML graph and general CUDA backend
SGLang	Native mixed NVFP4 checkpoint	Framework DFlash2 or MTP	General model executor, scheduler and cache backends
Custom NInfer	Exact fused .ninfer NVFP4/FP8 image	Five-layer DFlash2, seven proposals	Static Qwen schedule, CUDA Graph and accepted-prefix GDN replay

The cache was the real bug

The first build completed successfully on the RTX PRO 6000. The converter ran, the image built and roughly 21 GiB of weights loaded in eleven seconds. Then the server failed before its model endpoint became ready:

kv_cache_append_prefix: invalid cyclic cache

The problem was not the total context length. Qwen3.8's DFlash2 layers use a 2,048-token local window, while the generic cache-append path had 4,096 encoded as a constant.

At first this looked like a validator problem. It was not. The CUDA kernel also used a 4,095 bit mask to choose the cyclic slot. Simply allowing the 2,048-token cache through validation would have moved the failure into incorrect GPU indexing, which is a much less friendly kind of bug.

I changed the operator to dispatch explicit 2,048- and 4,096-token specialisations and added wraparound, guard-region and CUDA graph-replay tests for both. The existing callers retained the old geometry, while Qwen's DFlash2 path received the window it was trained to use.

The unchanged server then started correctly.

Reproducibility has to constrain the build

The next issue was stranger. Our appliance manifest pinned both CUDA base images by digest, but the Dockerfile still referred to them by mutable tags. The preparation script pulled the correct digests and attached the expected tags locally, yet BuildKit resolved different registry metadata during the build.

In other words, the manifest could claim one CUDA base while the resulting image contained another.

The fix was to generate a reviewed Dockerfile containing literal digest references, reject any remaining mutable CUDA FROM line and record the generated file's SHA-256 in the appliance manifest. A reproducible build is not one which lists exact versions somewhere. Those versions must actually constrain what gets built.

What happened on Blackwell

The corrected path passed its real CUDA tests on an NVIDIA RTX PRO 6000 Blackwell Server Edition, including both cache geometries and graph replay. It survived two cold starts, returned the exact model identity, produced the deterministic answer 42, followed a strict JSON schema and emitted a correctly typed tool call.

The direct completion reached 158.7 generated tokens per second, accepting 4.20 tokens per speculative round. The constrained tool call reached 226.5 tokens per second, accepting 6.10 tokens per round.

This difference is important. Speculative decoding is faster when the draft model can predict more of the continuation. A constrained tool call is easier to draft than open-ended generation. We crossed 200 tokens per second on a real admitted path, but I am not presenting that as the universal speed of the model.

The wider six-arm evaluation against llama.cpp, SGLang and vLLM remains unfinished, so the production flag remains off. As with the model research I wrote about previously, the strongest intermediate number is not automatically the final result.

Why this matters for MLX

There is a version of the open-model argument which says that model weights are becoming commodities and therefore the infrastructure underneath them will matter less. I think the opposite is happening.

As the weights improve, more people can start from a capable model. The advantage moves into quantisation, kernels, speculative decoding, memory layout, serving, evaluation and the ability to prove exactly what ran. Closed APIs hide those decisions behind a token price. Open models expose them as an engineering surface.

This is why self-hosting matters for Project Venus. It is not simply that the model can run on infrastructure we control. We can change the cache operator when its assumptions do not match the architecture, constrain the complete build by digest and evaluate the result against the same real task.

Alibaba gave us the model. NVIDIA gave us the low-precision hardware. The remaining performance had to be earned in the path between them.

---

Research context: China Is Winning the Post-Transformer Race. Why open Chinese labs are releasing architectural progress that the rest of us can build upon.

Architecture: LLM-Agnostic by Design. Why model portability only matters when every candidate faces the same governed test.