# Three Cents of Probing Says Jev Is a Small Qwen in a Very Good Suit

> Black-box fingerprinting TypeSafe's decision model: what token counts, latency curves and error messages give away about the trunk underneath.

Published: 2026-09-18T10:00:00Z
Updated: 2026-09-18
Author: George Pullen (Author, MLX)
Category: Research
Tags: jev, typesafe, system-one-models, qwen, model-fingerprinting, rlcd, calibration, open-weights
Canonical URL: https://mlx.systems/blog/fingerprinting-jev-for-three-cents

## TL;DR

- A ~$0.03 black-box probe of Jev found a pretrained multilingual LLM trunk of roughly 3 to 4B active parameters, a 32,768-token context window, and a customised tokenizer that matches no stock vocabulary.
- The serving pattern is one shared prefill over the state with the option set scored in one batched pass against it: flat latency from 5 to 240 options, no order effect I could measure, and a hard 255-choice cap.
- The genuinely interesting part is the training, not the architecture: confidence values that are calibrated against realised accuracy (RLCD) are reproducible on an open-weight trunk with executable outcomes as labels.
- For anyone already serving open Qwen models, every load-bearing component of Jev is buildable in-house.

---

The tell was the commas. A hundred commas and a hundred quotation marks each cost Jev *exactly* the same fifteen tokens on the billing meter, and no tokenizer on my disk treats those two characters alike. Qwen bills them differently. GPT-4o's o200k bills them differently. Llama bills them differently. That flat equality was the first hard evidence that the vocabulary inside [TypeSafe's new decision model](https://typesafe.ai/blog/introducing-system-one-models-and-jev) belongs to nobody's stock model, and it cost me about a cent to find.

![Three tokenizer lanes: Qwen splits commas into single tokens, o200k merges them into long runs, and Jev bills commas and quotes identically, the tell that no stock vocabulary matches.](/blog/fingerprinting-jev/tokenizer-tell.svg)

Jev launched last week on a bold pitch: no text generation, just typed, schema-constrained decisions with calibrated probabilities, trained with something they call RLCD, claimed to be up to two hundred times faster than frontier LLMs at comparable intelligence. We serve open Qwen models for structured analytical work, so the question that mattered to me was blunt: is there new science under this, or a serving pattern I could build myself? Black-box probing answers that kind of question cheaply, because a metered API has to publish its meter.

## The method: bill me in tokens and you've told me your tokenizer

Jev returns `usage.input_tokens` on every response. It bills $0.042 per million input tokens, so it has to. The prompt template is invisible, but it is constant, so it cancels: measure `tokens(state A) − tokens(state B)` through the API, compute the same difference locally under each candidate tokenizer, and match. On adversarial strings (digit runs, punctuation runs, CJK, Cyrillic), token *counts* are nearly as identifying as token identities.

Five independent numeric probes (30-digit runs, π's digits, dates, comma-grouped numbers) matched Qwen's single-digit splitting exactly, where GPT and Llama vocabularies group digits and disagree. Six punctuation-run probes matched o200k exactly, where Qwen disagrees. And natural Chinese, Cyrillic and Korean came back at roughly one token per character, which is *worse than every major tokenizer*. No stock vocabulary does all three. I pulled Qwen3.6's new 248k vocabulary to check whether "modified" might just mean "newer generation": it matches even less. The tokenizer is custom, probably pruned and retrained around a JSON-heavy decision workload, which also happens to be the honest economic move, because a 248k-entry embedding table is real serving cost for a workload that never touches rare CJK merges.

## Small trunk, real prefill, parallel scoring

Bisection found a hard context limit of **32,768 tokens**: the API literally errors `max_tokens_exceeded` past it. Every frontier model is 128k+. That single number moves Jev from "mysterious new architecture" to "sub-10B open-weight class".

The behavioural probes then confirmed what that class implies. Knowledge has a genuine pretraining gradient (April 2024 facts at 0.70 confidence, January 2025 at 0.56, August 2025 leaning false at 0.19, September 2026 news at a coin-flip 0.52), which bounds a cutoff in early-to-mid 2025. Chinese, Japanese and German inputs with English rubrics all came back at probability 1.00. A two-step arithmetic word problem scored 0.99. An "ignore all instructions" injection moved a false-statement probability from 0.04 to 0.23 without flipping it: trained resistance sitting on an instruction-following substrate. This is a pretrained multilingual LLM, as close to certain as black-box work gets.

Latency gives the size. Upstream service time ran 54ms at 2,300 tokens to 203ms at 31,000: real prefill, scaling with input, with a buried fact retrieved at 0.99 from the start, middle and end of a 30k-token state (full attention, no truncation). The marginal throughput works out to roughly 1 to 4B active parameters on an H100 at bf16, maybe 8B at fp8. My best guess is 3 to 4B active, and a 30B-class mixture-of-experts with ~3B active fits every measurement, including the quality.

The scoring pattern is the pretty part. Latency is flat from 5 options to 240; option order shifted nothing in my probes (all saturated cases; an independent investigation with competitive option sets measures a small position effect, so I won't claim perfect invariance); output accounting runs ~9.6 token-units per option; and past 255 options the API refuses with a refreshingly honest `"Too many choices. Must have at most 255 choices."` That is one shared prefill over the state with the option set scored in a single batched pass against the cached representation: constrained decoding industrialised, exactly the pattern a hobbyist can approximate on a laptop and a company can serve with output tokens priced at free, because there is no decode to pay for.

One debt of honesty: [Archer Hume's wider campaign](https://archerhume.com/posts/jevs-architecture-unmasked) went up the day before this post, and it agrees with the architecture verdict above while going further on calibration (a measured MMLU ECE of 0.031) and on option interactions, which his tests show are listwise rather than strictly per-option independent. Two investigations, one answer.

## What's actually novel

Strip the launch language ("new architecture, new sampler, new objective") and the trunk is ordinary; the sampler is known technique, superbly engineered; and the objective, RLCD, is the published calibration-reward family under a new name. What is genuinely good is the product insight: typed probabilistic decisions as the primary contract between software and a language model, with economics that follow from deleting the decode. And one measured detail suggests the calibration is real work, not branding: on a baseline call the top probability was 0.84 while reported confidence was 0.596, a learned calibration map, not a restatement of the argmax.

Everything load-bearing here is reproducible on an open trunk, which is rather the point for us. If a decision layer over Qwen with calibrated scoring can be trained against outcomes your own systems can verify, the interesting question stops being "what's inside Jev" and becomes "whose calibration do you trust: one graded against two frontier models' averaged opinions, or one graded against executable ground truth you can audit?" I know which bet I'd take.