One APU, one week,
a 97.5× serving turnaround

§0 · Key findings

Past the GGUF baseline under real serving load — with working tool calls and vendor-quality weights

The native ROCm stack loses the single-stream footrace and wins everything that makes a serving stack: aggregate throughput, tool calling, calibrated weights, and a codebase that can be fixed — which is exactly what happened all week.

All results below are matched-conditions, fresh-service confirmed. Raw decode lands at 21.44 tok/s under strict TG32 with exact outputs; the native multi-token predictor lifts effective decode to 26.23 tok/s at 82.35% acceptance with zero faults. Prefill holds 521 tok/s @512 on the current build, with 540/550/536 tok/s measured at 2K/8K/16K on the earlier 14.84-era build — treated as a floor. The honest reference point is our own llama.cpp Vulkan GGUF build: 45.24 tok/s raw, 51.08 with MTP, 546–591 tok/s short prefill. Single-stream, the native stack runs at ~51% of that decode speed. Under real serving load it pulls ahead.

§1 · The model & the machine

A 124B MoE that fits in one APU's memory — if the format is right

Three design facts decide everything downstream: a hybrid KDA+MLA attention stack, a single native MTP head, and an official int4 checkpoint that is the only format which fits 128 GB of unified memory.

The stack, full pins

§2 · The landmine map

How 0.269 became 26.23 — seven gated steps, 97.5×

Each step was measured in isolation, exact-output verified, and promoted only past its gate. The log scale is the honest one: the campaign is a sequence of multipliers, not increments.

Two diagnostics carried the week. Before any profiling began, all 99 warmup autotune events were exonerated — JIT was never the problem. The decode collapse was then isolated by a 5-step torch-profiler trace, and the MTP memory fault was bisected to a context-length boundary: healthy at 4,034 tokens with 83.3% acceptance, faulting at 4,101.

The fault, bisected to a page boundary

The MTP memory fault was not a mystery: context length 4,034 tokens runs healthy at 83.3% acceptance; 4,101 faults. The line between them is the third 2048-token attention page, where multi-split attention activates and merge_attn_states_kernel proves unsafe on gfx1151 (grid [640,32,1]). Safe-merge plus single-split verification routes around it.

§3 · Real serving load

Where the native stack passes the baseline: four concurrent requests

Single-stream, the GGUF build is faster. Serving is not single-stream — aggregate decode crosses the baseline's best at C4, with per-request decode still at 15 tok/s per user.

Aggregate token generation scales from 26.33 tok/s at C1 to 62.16 tok/s at C5 (2.36×), while prefill aggregate holds 503–508 tok/s through C4. The GGUF baseline's single-stream best — 51.08 tok/s with MTP — is crossed at C4. The ceiling is stated plainly: at C6 the engine fails with a documented high-concurrency HIP crash class seen on large MoEs on this APU, so serving is capped at 5 pending root-cause.

§4 · MTP depth study

Depth 1 is the answer for single-NextN-head models

Position 1 accepts at 82.35% — the head doing what it was trained for. Position 2 chains the head on its own out-of-distribution output and falls to 27–33%. Every extra position still pays a full pass through the 3.1B draft head.

Measured per-position draft acceptance makes the arithmetic stark: on a bandwidth-bound APU, speculation past the first token is priced out. K1 nets +27.3% against K2's +20.2%, and K3 is structurally unfavorable. The guidance travels: if you are running Ling, Qwen3-Next, or any single-head MTP model on edge hardware — depth 1.

§5 · The gfx1151 reality table

What doesn't work on this silicon — measured, so you don't have to

Every row is matched-conditions. The biggest measured cost block — the BF16 KDA projection GEMMs at 4.41 GB/token — attracted three quant-kernel attempts; all three were closed without wasting a calibration run.

The structural row sits at the bottom: gfx1151 has no FP4/FP8 matrix instructions (F16/BF16/IU8/IU4 WMMA only). MXFP4 anything is a dequant dead end on RDNA — int4 weight-only is the right format here, and the unused IU4 WMMA path is the only silicon-level headroom left on this APU.

§6 · The verdict

Run native — even single-stream

The GGUF build wins the footrace and loses the job: it fails agent tool calls, ships a third-party weight conversion, and is a llama.cpp silo. The native stack is the one that can be fixed — and was.

We run the native stack anyway, even single-stream: the vendor's calibrated checkpoint, structured tool calling via inclusionAI's own ling3 parser, OpenAI-compatible serving with continuous batching, and a codebase we can fix is worth more than the footrace. The levers left on the table are specific: KDA projection GEMMs running ~137 GB/s against ~182 GB/s sustained on the largest shapes, a selective qkvb-only W4A16 quant that went 1.479× faster in the exact-kernel screen, and the C6 engine failure to root-cause next.

Upstream

Findings being filed or referenced: the vLLM ROCm skinny-GEMM pathology on wave32 (28.16×); the merge_attn_states_kernel HSA memory fault on gfx1151 with the page-boundary bisect; the compiled multi-token verifier falling off optimized MoE paths; the gfx1151 dense W4A16/W8A16 kernel reality table; and the C6-class concurrency engine failure once root-caused. Related issues: vllm#45871 (wave32 sampler), vllm#37035 (GDN MTP crash class), vllm#50264 (the RDNA decode-collapse attribution method we reused).

Reproduce

Start from vLLM main ≥ 2026-08-05 (PR #51045), TheRock gfx1151 nightlies (torch 2.13 / HIP 7.15 / Triton 3.8), the config pins in §1, and the patch set from the ladder — every step is gated, measured, and revertible in the campaign record. Baselines measured against: hec-ovi/laguna-s-2.1-vllm-strix, hec-ovi/vllm-awq4-qwen, kyuz0/amd-strix-halo-vllm-toolboxes, AMD's TheRock nightlies, and AMD-AGI's TraceLens for offline trace attribution.