China Is Winning the Post-Transformer Race

When I say that I think we should be more bullish on Chinese AI labs, it has nothing to do with politics at all. If we are talking purely from a researcher's point of view, or indeed anyone who values transparency, it's not particularly difficult to see why some would prefer the likes of Moonshot, Alibaba or Zhipu to dethrone OpenAI and Anthropic.

These labs are taking architectural risks that the US labs are not taking in public, and then releasing the results for other researchers to build upon. Kimi Delta Attention is the clearest example, and in my opinion represents the biggest serious move away from the original Transformer architecture introduced by Vaswani et al. in their 2017 paper, Attention Is All You Need.

What Kimi Delta Attention actually changes

If your understanding of attention starts and ends with the original Vaswani paper, the distinction is fairly simple. Normal attention retains a key and value for every previous token, and each new query searches that history:

$$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^{\mathsf T}}{\sqrt d}\right)V$$

The model does not have to compress its history, but the cache grows with the context.

KDA instead carries a matrix, $S_t$, through the sequence: a small associative memory that the model edits as it reads.

$$S_t=(I-\beta_t k_tk_t^{\mathsf T})\operatorname{Diag}(\alpha_t)S_{t-1}+\beta_t k_tv_t^{\mathsf T}$$

The term on the right writes a new key-value association, while the rest decides what should be retained or corrected. Gated DeltaNet did this with one forget gate per head; KDA gives every channel its own gate, allowing one part of the state to be preserved while another is overwritten. This makes the same basic idea far more expressive.

The fast-weights interpretation is the part I find most interesting, because the model is changing a temporary weight matrix while it reads. This suggests future architectures may actually learn at test time, although KDA is not itself a test-time-training objective. TTT optimises a self-supervised loss during inference, whereas KDA applies an update rule learned during pre-training. The similarity is real, but they are not the same thing.

Why sparse attention is still necessary

Compressing the context into a fixed-size state inevitably loses some information. This is why Kimi Linear still uses one full-attention layer after every three KDA layers, and why GLM-5.3-Flash follows almost the same pattern using sparse attention. Its 45-layer stack contains 34 KDA layers and 11 DeepSeek Sparse Attention layers, so for now the post-Transformer architecture remains a hybrid.

GLM-5.3-Flash was initially released anonymously as Ox Alpha, meaning people saw it perform serious agentic research work before they knew it came from Zhipu. It has beaten Fable on individual autoresearch and kernel-optimisation tasks, although Fable remains ahead overall on the public kernel board. The important point is that an open model of this size is now competitive on work that recently appeared to be the preserve of closed frontier models.

Why this matters for MLX

This is directly relevant to Project Venus. We already run Alibaba's Qwen models through infrastructure we control, and intend to use more open models across MLX. Sovereignty and self-hosting matter, but so do constrained decoding, llama.cpp, multi-token prediction and the ability to optimise the entire system ourselves.

I am also interested in the internal model signals. KDA gives us a changing memory state, while sparse attention indicates which parts of the context were retrieved. Neither is provenance, but both may help us understand why a model produced a particular answer and build better receipts around it.

OpenAI and Anthropic still make exceptional models, and we will continue to use them. I am bullish on Moonshot, Alibaba and Zhipu because they are releasing more of the architectural future in a form we can inspect, run and improve ourselves. For the systems we want to build at MLX, that matters more than who has the best closed model at any given moment.