# Inside Inkling 2 Mini

> How Thinking Machines dethroned the king of local AI using a Looped Transformer.

Published: 2026-11-21T19:00:00Z
Updated: 2026-11-21
Author: George Pullen (Research Engineer, Mercury Labs)
Category: Research
Tags: inkling-2-mini, thinking-machines, attention-residuals, looped-transformers, interpretability, project-venus
Canonical URL: https://mlx.systems/blog/inkling-2-mini-notes

---

Yesterday, Thinking Machines [released Inkling 2 Mini](https://thinkingmachines.ai/news/inkling-2-mini/). A model that outranks Qwen3.8 27B on DeepSWE and on SWE-bench Pro, and has only 3.4B active parameters (5.4B weight activations per token once looping is counted). Meanwhile, Qwen3.8 activates all 27B of its parameters each forward pass.

A big reason they were able to pull this off was their adoption of a 'Looped Transformer' architecture. However, they do not simply feed the hidden state from the previous forward pass back into layer 1 for the next iteration, it's much more sophisticated than that.

Main architectural innovations:

- Looping middle layers
- Attention Residuals

The 24 layers in Inkling 2 Mini are split into four blocks: **\{ B₁, B₂, B₃, B₄ \}**. The residual output of B₁ (layers 1-6) represents a solid foundational understanding of the input embeddings. Essentially the product of taking the embeddings and computing the basic syntactic and semantic connections between words, but nothing too complex or nuanced is learned at this stage.

B₂ and B₃ are the looped layers, which are invoked 3 times for every token the model generates. I like to think of this as the model's working memory, where the understanding compounds by looking at what was learned last time around, and building on it, increasing depth and expressivity but not adding any additional parameters. Once the loop has run 3 times, the residuals from the third loop's B₂ and B₃, along with the original B₁ (which is crucially preserved as a reference to the initial information), go into B₄, which is where this understanding is decoded into a projection of predictions for the next token.

Essentially, the first block takes the input and turns it into something that's actually representative of its meaning. The middle two blocks take this transformed input and, through 3 loops, enrich it to fully capture all nuances. Then the fourth block takes this rich representation and converts it into something that's actually useful for predicting what the next token is going to be.

[Goodfire's preliminary findings](https://www.goodfire.com/research/interpreting-inkling-2-mini):

- layers coordinate with each other, and with past/future versions of themselves
- they, counterintuitively, choose to write to themselves and read from future layers

## Attention Residuals

Conventionally, you have:

![Handwritten: h = x_l + Attn(LayerNorm(x_l)), then x_(l+1) = h + MLP(LayerNorm(h)).](/blog/inkling-2-mini/01-conventional-block.png)

But, with depth-wise attention ([Attention Residuals](https://arxiv.org/pdf/2603.15031), Kimi Team):

→ use a weighted mixture of earlier blocks' residual streams

For Inkling 2 Mini:

![Handwritten: B_1 = sum over i = 1 to 6 of (a_i + m_i), B_2 = sum over i = 7 to 12 of (a_i + m_i), B_3^Partial = sum over i = 13 to 15 of (a_i + m_i), where a_i is the attention output and m_i the MLP output.](/blog/inkling-2-mini/02-block-sums.png)

The input to Layer 16's Attn is:

![Handwritten: x_16^A = alpha_0 e + alpha_1 B_1 + alpha_2 B_2 + alpha_3 B_3^Partial, with the alphas annotated as attn weights, worked out per-token.](/blog/inkling-2-mini/03-layer-16-attn-input.png)

Each sub-layer gets a learned vector:

![Handwritten: w_16^A in R^d](/blog/inkling-2-mini/04-pseudo-query.png)

*represents layer 16's learned vector for its attention sub-layer*

V is a matrix of the stored block states:

- V₀: embedding (e)
- V₁: block 1, layers 1→6
- V₂: block 2, layers 7→12
- V₃: block 3 partial, layers 13→15

Vᵢ represents which of these we are referring to.

## Scoring Equation

![Handwritten: S_i = w_16^A transpose times RMSNorm(V_i); so S_0 = w_16^A transpose times RMSNorm(e), through to S_3 = w_16^A transpose times RMSNorm(B_3^Partial).](/blog/inkling-2-mini/05-scoring.png)

## Then apply Softmax

![Handwritten: [alpha_0, alpha_1, alpha_2, alpha_3] = Softmax([S_0, S_1, S_2, S_3])](/blog/inkling-2-mini/06-softmax.png)

![Handwritten example: [alpha_1, alpha_2, alpha_3, alpha_4] = [0.1, 0.2, 0.5, 0.2], therefore x_16^A = 0.1e + 0.2B_1 + 0.5B_2 + 0.2B_3^Partial.](/blog/inkling-2-mini/07-softmax-example.png)

Depth-wise attention makes a lot of sense intuitively. Instead of each sub-layer having to consume a firehose of information, it instead receives a much more targeted and filtered down version of it that highlights what's actually relevant to it. When it comes to a Looped Transformer such as Inkling 2 Mini, that sort of filtering becomes crucial to ensure that each layer can continue to enrich the hidden representations, without compromising the ability of future layers to see what's important to them. If we think about the input to block 4, not only does it receive the enriched representations from blocks 2 and 3, which have been run 3 times round, but it can also go back and cross reference that with the much simpler representation that B₁ produced, and it can even go back and look at the original embeddings.

## Depth-Wise Attention with Cross-loop Attention

![Handwritten. Depth-wise: x = sum over i = 0 to 3 of alpha_i times V-hat_i, where V-hat_i = RMSNorm(V_i). Cross-loop: M = beta_2 B-hat_2^[t-1] + beta_3 B-hat_3^[t-1]. Notice: B-hat_1 is not included as it is the 'prelude' block.](/blog/inkling-2-mini/08-depth-wise-cross-loop.png)

![Handwritten: x' = kappa ⊙ x + gamma ⊙ M, with kappa and gamma in R^d. x is the current loop, M the previous loop, kappa the keep gate and gamma the gain gate.](/blog/inkling-2-mini/09-keep-gain-gates.png)

### @ Loop 3

![Handwritten: M^[3] = beta_2^[3] B-hat_2^[2] + beta_3^[3] B-hat_3^[2]](/blog/inkling-2-mini/10-loop-3-message.png)

→ Therefore, loop 3 cannot attend to information from loop 1

## Important Detail

Loop 2 **does not** start with the final residual from Loop 1. Instead, it goes back to B₁.

This means B₁ acts as the stable representation of the input, whilst B₂ and B₃ (from the previous loop) carry intermediate representations like working memory.

---

Overall, I think the success of Inkling 2 Mini shows that we can pack much more depth into the same amount of parameters than previously thought. This is a positive signal for local AI, and hopefully there will soon be many more entries from OSS model labs in this space.

For Project Venus and MLX, this model is an opportunity to explore what happens when you have Qwen3.8 27B intelligence in only 3.4B active parameters, and that could mean a much more cost-effective way to host agent swarms such as [Venus Colony](/research/colony).

## References

- Thinking Machines, [Inkling 2 Mini](https://thinkingmachines.ai/news/inkling-2-mini/), 20 November 2026.
- Goodfire, [Interpreting Inkling 2 Mini](https://www.goodfire.com/research/interpreting-inkling-2-mini), 20 November 2026.
- Kimi Team, [Attention Residuals](https://arxiv.org/pdf/2603.15031), arXiv:2603.15031, 2026.
