boxwrench/Ling-3.0-flash-int4-wrench

Ling-3.0-flash — Wrench INT4

An INT4 compressed-tensors S4/g32 quant of inclusionAI/Ling-3.0-flash, using the official INT4 tensor layout, configuration and tokenizer. Weighted toward agentic coding and tool use, with a general-text floor.

Base model	Ling-3.0-flash (MIT); architecture
Runs with	vLLM; measured build 0.26.1rc1.dev467+g838616875 (ROCm)
Tested fit	8060S / Max+ 395, 128 GB unified memory; check HF file sizes before downloading
Full results, method, reproduce	Ling-3.0 page

Results

Model	Status	Prompt	Prefill tok/s	Decode tok/s	TTFT (s)
Wrench	SERVES OK	1K	961.4 [949.6-962.1]	27.86 [27.36-29.70]	1.09 [1.09-1.10]
Wrench	SERVES OK	8K	1083.0 [891.1-1084.2]	27.00 [26.99-28.70]	7.58 [7.58-9.22]

Measured 2026-10-17 on Radeon 8060S, Ryzen AI Max+ 395, 128 GB unified; Linux/ROCm, vLLM build above. Greedy, 64 generated tokens, MTP enabled; three-run medians [min–max], nonce per request. Keep ranges beside comparisons. Serving is not a quality measurement; no task-accuracy claim. Receipt and limits.

Quality

Metric	Official INT4	Wrench
KL to FP8 ↓	0.1402	0.1230
Top-1 agreement with FP8 ↑	92.61%	93.28%

Against Ling-3.0-flash-fp8 on a fixed 92 × 4096-token corpus; not task accuracy. Per-domain table.

Run it

Prepare a compatible vLLM installation with INSTALL. Measured on weight revision f6eb44cf8192. The vLLM build was a ROCm build, not an upstream release, so compare checkpoints on one server rather than absolute numbers across builds. Untested download and launch example; set MODEL to a local directory and review checkpoint code before trusting it:

hf download boxwrench/Ling-3.0-flash-int4-wrench --local-dir "$MODEL"
vllm serve "$MODEL" --host 127.0.0.1 --port 8000 \
  --attention-backend TRITON_MLA --moe-backend triton \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --max-model-len 262144 --gpu-memory-utilization 0.72 --max-num-seqs 6 \
  --max-num-batched-tokens 8192 --enable-chunked-prefill --enable-prefix-caching \
  --mamba-cache-mode align --trust-remote-code

Discrete GPUs, SGLang and CUDA are untested. Agent setup: AI_SETUP.

Your numbers

How to measure and report.

Credits

inclusionAI for the base model and official INT4 (MIT); vLLM for serving (Apache-2.0). Licence notices.