TL;DR: We trained a DSpark speculative-decoding draft head for GLM-5.2-NVFP4, pushing accepted length beyond baseline.

• Four targeted changes: layer-78 final residual tap, per-layer tap fusion, position-specific learned queries, and a recurrent head.

• 50K-sample ablations: validation accepted length rose from 3.517 to 3.797.

• Scaled training to 500,857 on-policy examples (block-16): reached τ = 4.915 by epoch 9.

Large language models generate one token at a time. That makes decoding the expensive part of serving: each new token requires another pass through the target model.

Speculative decoding changes the rhythm. A small draft model predicts several tokens ahead, then the target model checks that proposal in parallel. When the draft is accurate, the target model can move farther forward on each decoding step [1]. This is called accepted length, τ: higher is better because each target-model verification step produces more tokens.

We trained a DSpark draft head [2] (a speculative decoding method that drafts tokens using a semi-autoregressive architecture) using the Speculators library for GLM-5.2-NVFP4, and focused on one question:

Is it possible to improve DSpark accepted length even further?

The answer is a set of four complementary changes, validated first on 50k samples and then trained at 500k scale.

The result: validation accepted length increased from 3.517 for Baseline DSpark to 3.797 for the Full Recipe when trained with 50K training examples. When scaled up to 500,857 on-policy training examples and a block-16 layout, the final checkpoint reached τ = 4.915 as opposed to DSpark head from RedHat with 3.96 accepted length.

Dataset preparation and on-policy data generation

The draft must learn to follow the target model, not a generic text corpus. We therefore first prepared a prompt-only dataset and then generated every training response with GLM-5.2-NVFP4 itself.

• 1.93M source prompts across code, math, chat, retrieval-augmented generation, tool use, and long-context tasks.

• 500,857 generated responses in the final on-policy training set. These were produced from the source prompts using GLM-5.2-NVFP4 at high reasoning effort, with the remaining prompts reserved for future training.

• A 16,384-token generation limit as well as keeping the truncated examples.

Our proprietary dataset is designed to be broad: code and math account for just over half of the examples, while chat, RAG, tool use, and long-context prompts keep the draft from overfitting to a single workload.

Four changes that made the draft stronger

We included several changes to DSpark to increase the acceptance rate. These changes were first validated on 50K training samples for five epochs before large scale training on 500K samples.

Training on 50K samples. Each individual change modifies one component of Baseline DSpark and is compared directly with it; the individual changes are not measured cumulatively in this figure. The Full Recipe combines all changes.

1. L78 Tap + GLM projection init: Give the draft GLM's final residual state

Baseline DSpark reads five intermediate hidden states from GLM. We added the final residual state from layer 78 as a sixth input.

HyperDFlash [4] motivated this change by reusing the final residual already provided to DeepSeek's native MTP path. The GLM analogue is layer 78 and GLM's native MTP projection (eh_proj), which already reads that final residual. We added layer 78 as a sixth input to the shared fc projection that fuses the GLM taps, and used the layer-78 part of eh_proj only to initialize the new layer-78 portion of that projection. The projection still combines layer 78 with the five intermediate taps; it does not use layer 78 alone.

This was the largest single gain: 3.517 → 3.698 at epoch 5. The effect is strongest at the first speculative position, which is where a final-layer representation should help most.

2. Per-Layer Fusion: Each draft layer decides which GLM taps it needs more

Baseline DSpark combines five GLM taps into one context immediately, then shares that context with every draft layer. Per-layer fusion, following DFlare [3], keeps the five contributions from the shared projection separate for one extra step. Each draft layer then chooses its own five-way mixture. Every draft layer still uses all five contributions; only the relative emphasis differs.

The large fc projection remains shared, with no additional target-model forward pass. This adds just 25 scalar logits in the five-layer, five-tap run (30 in the six-tap Full Recipe). The logits start at zero, so every layer initially uses an equal mixture, equivalent to Baseline DSpark after RMSNorm.

Per-layer fusion keeps the shared projection's five contributions separate for one extra step. Each draft layer then assigns its own weights to all five before context K/V is built.

This change reached 3.588, a +0.071 gain over Baseline DSpark.

3. Position Embeddings: Give each future slot its own starting query

In Baseline DSpark, every unknown future slot starts from the same mask-token embedding. RoPE still tells attention where each slot sits in the sequence, but the slots otherwise begin with the same query.

We gave each future slot its own learned starting query. Slot 1 receives query 1, slot 2 receives query 2, and so on. RoPE remains unchanged; the learned table complements it by letting the model ask a different question when it is predicting farther into the future.

RoPE is active in both cases. The learned table changes only the starting query for each future slot.

For block 8, this is a small table of 43,008 values 7×6144; block 16 uses 92,160 15×6144. It adds no matrix multiplication. Every row starts as a copy of the original mask embedding, so training begins from the baseline behavior. This change reached 3.677 (+0.160), with the largest gains in the middle and late positions.

4. Recurrent Head: Carry state across the speculative block

Baseline DSpark's Markov head sees only the immediately preceding drafted token. We used the small recurrent state, allowing later positions to use a summary of the entire draft prefix.

This adds 5.1M parameters and reached 3.687 (+0.170).

The Full Recipe

The four changes work together, although their gains are not additive. They address some of the same errors, so the final improvement is smaller than the sum of the individual improvements.

Scaling from 50k to 500k examples

After the ablations, we trained the Full Recipe on 500,857 on-policy examples with block 16.

The first three epochs produced the largest jump, but validation acceptance continued to improve through epoch 9. From epoch 1 to epoch 9, the model added 1.218 accepted tokens per verification step.

Speculative decoding is just one lever that makes Makora Inference super fast. Our automated optimization stack spans kernels, inference server tuning, and new algorithms.. We're continuing to train our DSpark draft head to push accepted length even further.

References

1. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023.

2. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv:2607.05147, 2026.

3. DFlare: Scaling Up Draft Capacity for Block Diffusion Speculative Decoding. arXiv:2606.02091, 2026.

4. HyperDFlash: Hyper-Connection-Aligned Block Speculative Decoding with Gated Residual Reduction. arXiv:2606.26744, 2026.