AngelSlim/Qwen3-8B-DFly-Block8

A more accessible, comprehensive, and efficient toolkit for large model compression.

✒️ TechnicalReport   |    📖 Documentation   |   🤗 Hugging Face   |   🤖 ModelScope

💬 WeChat |   🫨 Discord

Qwen3-8B No-Thinking Drafters: MTP and DFly

Model Summary

This repository contains speculative-decoding drafter checkpoints for Qwen3-8B, including an MTP drafter and a DFly drafter. The drafters propose candidate tokens for the target model to verify, improving inference throughput while preserving the target model's output distribution.

The models were trained with AngelSpec and are intended to be deployed in vLLM together with the Qwen3-8B target model.

• Target model: Qwen3-8B
• Inference mode: No-thinking
• Drafter types: MTP and DFly
• Training framework: AngelSpec
• Recommended serving backend: vLLM speculative decoding
• MTP speculative tokens: 3
• DFly speculative tokens: 8

Training Data

For Qwen3-8B, prompts are taken from Open-PerfectBlend (Xu et al., 2024). Responses are regenerated with the Qwen3-8B target model in non-thinking mode, and the resulting target-generated responses are used to train the drafter.

Regenerating responses with the same target model helps the drafter better match the target distribution, which is critical for speculative decoding acceptance rate and end-to-end acceleration.

Training Hyperparameters

These Qwen3-8B draft models use the following training configuration:

Hyperparameter	Value
Batch size	512
Loss schedule	LK loss cold start for 300 steps, then end-to-end TV loss
Learning rate	6.0e-4
Minimum learning rate	6.0e-5
LR decay style	Cosine
Warmup ratio	0.04
Weight decay	0.0
Epochs	10
Maximum sequence length	4096

Evaluation

The table reports acceleration-related metrics across math, code, and chat benchmarks.

Target Model	Drafter	Math500	GSM8K	HumanEval	MBPP	LiveCodeBench	MT-Bench	Avg.
Qwen3-8B	MTP	3.53	3.56	3.33	3.22	3.25	2.57	3.24
Qwen3-8B	DFlash	4.97	5.54	4.77	4.50	4.46	3.16	4.57
Qwen3-8B	DSpark	5.87	6.25	5.56	5.25	5.20	3.77	5.32
Qwen3-8B	DFly	6.06	6.42	5.60	5.34	5.36	3.67	5.41

DFly achieves the highest average result among the listed Qwen3-8B drafters in this evaluation, with an average score of 5.41.

Deployment

Set TARGET to the Qwen3-8B target model and DRAFT to the corresponding drafter repository or local checkpoint path. The relevant inference code can be found in PR https://github.com/vllm-project/vllm/pull/50246.

MTP-3

export TARGET="Qwen/Qwen3-8B"
export DRAFT="<path-or-hf-repo-to-this-MTP-drafter>"

vllm serve "$TARGET" \
  --tensor-parallel-size 1 \
  --speculative-config "{"method":"mtp","model":"$DRAFT","num_speculative_tokens":3}"

DFly-7

export TARGET="Qwen/Qwen3-8B"
export DRAFT="<path-or-hf-repo-to-this-DFly-drafter>"

vllm serve "$TARGET" \
  --tensor-parallel-size 1 \
  --speculative-config "{"method":"dspark","model":"$DRAFT","num_speculative_tokens":7}"

Technical Report

For more details, see the technical report:

AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
arXiv: https://arxiv.org/abs/2607.25852