Crown Citadel Group Ciru Inference Lab llm.ciru.ai / research

Crown Citadel Research Note

Dynamic Draft Proposal

Dynamic Draft is a server-side policy layer for llama.cpp speculative decoding. It can select draft settings from the actual tokenized prompt length, or preserve a fixed setting when calibration shows that fixed policy is stronger than a context breakpoint.

Qwen 3.6 35B ROCmFP4 MTP 262144 context 2048-token generations 2026-06-21

The Proposal

Speculative decoding already has the low-level machinery needed to accelerate generation: draft depth, draft acceptance thresholds, and MTP draft paths. The missing layer is a practical serving policy that applies the fastest calibrated draft setting for the current tokenized context. External clients often cannot know the final prompt token count after chat templating and server-side formatting.

Dynamic Draft moves that decision into the server. After tokenization and before generation, the server reads the exact prompt token count, looks up the calibrated speed frontier, and sets the request's speculative parameters. If no context-specific row beats the best fixed setting, DD falls back to that fixed setting; a valid DD policy should not be slower than the best static draft policy included in its candidate set.

DD invariant: selected(ctx) = fastest calibrated safe setting at that context; fallback = best fixed static setting

One-sentence claim

Dynamic Draft turns the measured speed frontier into a server-side runtime policy, so aggressive short context settings and conservative long-context settings can coexist without client-side guessing.

Reasoning

Problem Client guessing The server sees the final tokenized prompt; clients usually only see a pre-template request shape.
Observation Upper envelope DD is not a rival profile. It should select the fastest measured safe row for each tokenized context.
Mechanism 0-1 us Persisted DD rows measured policy decision overhead at 0-1 microseconds.
Core reasoning: the server is the only place that reliably sees the final tokenized prompt. Therefore, the draft policy should run after tokenization, not in a model card footnote or in client-side guessing logic.

Design

  1. Tokenize the request. The prompt length is counted after server-side formatting and tokenizer behavior are applied.
  2. Select the calibrated frontier row. The policy chooses the fastest safe setting for the prompt-token range, with the best fixed setting as fallback.
  3. Clamp to allocation caps. DD can lower draft depth per request but cannot exceed the server's configured draft allocation.
  4. Respect explicit overrides. Request-level speculative.* fields win unless the server is launched with a force policy.
  5. Record the decision. Generation metadata includes whether DD applied, the reason, prompt tokens, selected settings, and decision time.

Static Draft vs Dynamic Draft Frontier

The fair comparison is simple: hold the static draft setting fixed at n2,p0.0 for every context, then compare it against the fastest measured safe setting at that same context. DD is the server-side policy that applies that per-context frontier after tokenization.

Current conclusion: the measured frontier beats fixed n2,p0.0 at the lower contexts where a more aggressive draft setting wins, then ties it at longer contexts where n2,p0.0 remains the best measured setting.

Measured comparison

Rows use 2048-token generations. The 4k row comes from the short-context request-draft sweep at 3946 prompt tokens; the 64k-256k rows come from the long-context DD proof sweep.

Context Static fixed setting Static TG DD-selected setting DD TG Gain vs static
~4k n2,p0.0 120.80 tok/s n4,p0.25 141.77 tok/s +17.4%
64k n2,p0.0 74.85 tok/s n4,p0.0 87.94 tok/s +17.5%
128k n2,p0.0 54.61 tok/s n2,p0.0 54.61 tok/s tie
196k n2,p0.0 49.89 tok/s n2,p0.0 49.89 tok/s tie
256k n2,p0.0 33.51 tok/s n2,p0.0 33.51 tok/s tie

Calibration Note

Policy quality comes from calibration. DD must be built from the measured frontier, not from a stale hand-written breakpoint. If a context-specific row loses to n2,p0.0, DD selects n2,p0.0. If a more aggressive row wins, DD selects the aggressive row for that context.

For this Qwen 3.6 35B proof set, the visible value is at shorter contexts: n4,p0.25 wins near 4k and n4,p0.0 wins at 64k. At 128k, 196k, and 256k the current measured frontier ties the static n2,p0.0 baseline.

References

  1. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast Inference from Transformers via Speculative Decoding.
  2. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating Large Language Model Decoding with Speculative Sampling.
  3. Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. Better & Faster Large Language Models via Multi-token Prediction.
  4. ggml-org. llama.cpp HTTP Server README.