qwenTPU: a TPU inside Qwen's weights

Why would anyone put a TPU inside a language model?

Nobody needs a TPU inside Qwen: a real TPU is faster, cheaper and does not lose its arithmetic to a rounding error. So why do it?

Because the bird teaches you something

A language model is a pile of memory (state), a way to look things up (attention) and a way to compute (FFNs). If that is enough to build an exact matrix-multiply machine, then we learn how much of "intelligence hardware" is hiding in the architecture itself. We are studying the bird, not selling the wing.

Because exactness is a hard test

Models are sloppy by design. Making every integer survive bf16 rounding, normalisation and a couple of dozen layers of someone else's code is a very unforgiving way to find out what the architecture really does.

What we do not claim

Not the first machine in Qwen's weights (Qwenputer and QwenPU got there first). Not faster than a real accelerator, or than QwenPU at the op level. Not useful for production. It is an existence proof, with receipts.

Why a Gated DeltaNet head is already a dataflow array

A TPU keeps weights put, streams activations, and accumulates partial sums in place. DeltaNet state S is a matrix with exact row write and row read.

One-hot write = store a row

With α=β=1 and a one-hot key, S += k(v − Sᵀk)ᵀ replaces row i exactly. That loads a weight row, or stores a partial sum.

Multi-hot read = adder tree

A multi-hot query returns the exact sum of the chosen rows. The contraction over the inner dimension is one read.

Counts, not values

Products are bit-ANDs, so accumulators hold small integer counts that bf16 stores exactly. The weighted sum is a free linear form.

Watch the array run

This is the array's semantics in JavaScript: the same tokens, state rows, bit-plane counts and clamp chain as qwentpu/circuit.py. It is not a transformer simulation. The result is checked against a plain integer reference.

chain a 2nd layer speed

Weights W (rows = inputs, cols = outputs)

Input vector x (activation tokens)

Output tokens

Per-column heads · DeltaNet state

weight bit in W head (rows × weight bits)AND-product in accumulator (rows × act bit × weight bit)

READ → counts → adder tree

waiting for a READ token…

Edit weights and inputs by hand

Architecture

Qwen's own layers are muted for chip tokens. The appended TPU layers hold the array. The model's sampled token goes back in as the next layer's activation.

The whole story in seven steps. The stepper above runs exactly this.

Architecture: Qwen layers, TPU layers (W heads, AND-product FFN, accumulator heads, adder-tree FFNs, code FFN), lm_head, OUT token fed back — The layer stack is the datapath. There is no program counter and no instruction stream.

Adder tree: bit-plane counts to s_j to Horner steps to output code — Counts become the output without ever forming the wide sum y: Horner top-down, q digits per step, clamp only when the range needs it, last step merged into the code FFN.

Token stream of one matvec — One matvec once the weights are resident: KR `ACT` tokens, then NC `READ` tokens, which emit NC `OUT` tokens. Weights loaded once as `CELL(r,c,w)` tokens, one per cell. There is no flush: every vector rewrites all KR accumulator rows.

Fine print. Qwen l2-normalises the query and scales it by 1/√(n_q·128). A multi-hot query is therefore padded to n_q = 2^odd rows so that scale is an exact power of two. Otherwise bf16 rounds each count differently (a count of 7 comes back as 6.96875).

Tokens are the bottleneck (packing)

A language model takes one token per step, and every step is a full trip through every layer. Our first array spent one token per input row and one per output column: KR + NC tokens for KR × NC multiply-adds. That is a delivery truck making one trip per parcel.

Packing loads the truck. An ACT token now carries g activations and a READ token reads h columns and answers with one OUT token carrying h outputs. Tokens per matvec fall from KR + NC to KR/g + NC/h.

Unpacked versus packed token stream and the g head copies per column — Same 64 multiply-adds: 16 tokens unpacked, 4 packed (8×8, g = h = 4).

The price is real: every column now keeps g copies of its weight head and g accumulator heads, a layer only has 16 key heads, so you need more layers, and the vocabulary balloons because one ACT token can mean any of 8^g things. MACs per token go 4 → 8 → 16 on 8×8, 32 on 16×16, 64 on 32×32. Every packed configuration is exact in the float64 evaluator and on real Qwen (0 mismatches). Try the packed presets in the stepper above and watch the token counter.

Skip the muted layers (chip-only path)

Qwen's own layers are muted for chip tokens: their attention output is exactly zero and their MLP sees a zero vector. A layer that adds nothing to the residual stream is the identity. So why walk through 24 of them to get nothing?

ChipPath goes straight from the embedding to the appended TPU layers, then the final norm, then only the chip rows of lm_head. It returns the same bits as the full model (a test checks that) and never touches Qwen's cache, so there is nothing to restore afterwards. It is the shortcut through a building where every corridor you skip is empty.

Full model versus the chip-only path that skips the muted Qwen layers — Text still goes through everything. Only chip tokens take the bypass.

Bug museum

Two exhibits, both of the form "the answer is off by a hair". Neither was our arithmetic. Both were somebody else's rounding.

Two numerics traps: an irrational query scale and a split-K bf16 reduction

Exhibit A. Qwen normalises the query and scales it by 1/√128. Pad our multi-hot query to n_q = 2^odd rows and that scale is a clean power of two; otherwise a count of 7 comes back as 6.96875.
Exhibit B. On an L4, cuBLAS sliced the long dot product (split-K) for 288 and 320 token traces and rounded the partial sums to bf16. Setting allow_bf16_reduced_precision_reduction = False keeps the reduction in fp32.

v2: a TPU that runs a program

v1 needed the host to choose every chip token. In v2 the host builds one prompt (weights, input, entry point) and runs argmax; append. The model emits every READ, ACT and OUTK token itself, applies relu and clamp in its datapath, and halts. v1 is untouched.

Watch a digit classified, token by token

Real traces: a 15→15→10 network with 5-bit integer weights, run as a depth-2 program through the 16-wide model (20 chip layers). Every token below was checked against the integer golden model and the circuit IR; the numbers come from social/anim/trace_v2.json.

speed

Prompt (host-fed, data only)

The digit (8×8) and its 15 pooled inputs

Layer 1: 16 hidden activations (relu + clamp, 0..31)

Layer 2: logits (OUTK values)

How it works

v2 architecture: prompt, model, next token, host does argmax and append — The host is only `argmax; append`.

v2 token timeline for one digit — Host-fed prompt, then 64 model-emitted tokens, then HALT.

ReLU feedback: the sum becomes the next ACT token — relu and clamp happen where the adder tree already ends, at zero extra layers.

what is fixed in the weights versus chosen by the prompt — Compiled in versus chosen by the prompt.

One compiled model, programs chosen by the prompt

The same model (Dmax = 4, 4-wide, 3 chip layers) runs a depth-1, depth-2 and depth-4 network. The prompt picks the depth by choosing which block the input is written into.

the entry block selects the depth — Entry block = Dmax − depth. Cost scales with the program: 2·d·n tokens.

Results and limits

Fixed in the weights: array size and number format, the READ, value, READ cycle, chain-only topology, the schedule. No branches, loops or early exit.
No signed logits: outputs are relu-clamped to 0..31, so negative scores read as 0 and ties are possible.
Accuracy: the integer network scores on 540 held-out digits. The model reproduces the integer network exactly; the gap to the float network comes from training with 5-bit integer weights, inputs quantised to 0..12 and a fixed scale (no requantisation shift), not from the hardware.

What is new, and what is not

The crowded part is "a machine inside Qwen". The new part is the machine's shape.

	What it is	Control	Lives in DeltaNet state	A token is
Qwenputer Qwen3.5-4B	RV32IM CPU, plus an in-weights NPU	PC, fetch, decode	registers, code, data	an instruction
QwenPU / qwenGPU Qwen3.5-2B	CPU + 64-lane int8×int4 vector unit; SIMT GPU	the CPU's program; host-compiled kernel	vector regs, weight rows, caches	a vector-op step or warp-launch record
TinyTPU RTL, inspiration only	systolic array in hardware description	sequencer	(registers in silicon)	(not applicable)
qwenTPU this repo	weight-stationary dataflow array	none: the layer stack is the datapath	the weights and the partial sums	a dataflow step

New: a TPU-style dataflow machine with no PC or ISA; the DeltaNet multi-hot read used as a reduction; weights and partial sums both in state; bit-plane counting that keeps every value bf16-exact; layers chained through the model's own tokens.
Not new: putting a machine in Qwen3.5's weights, or running a neural network on it. The ballast-normalised RMSNorm, DeltaNet-rows-as-RAM and token-trace ideas come from Qwenputer.
Not claimed: faster than QwenPU at the op level (its matvec streams 64 lanes per token; mine does 4 to 16 MACs per token). The scaling is O(N) tokens for O(N²) MACs, not a speed record.
TinyTPU is the model for the dataflow idea (stationary weights, streamed activations, in-place accumulation). No code or RTL is shared, and our array is not literally systolic: all columns work in parallel through separate heads.

A TPU inside Qwen's weights.