Qwen3.6-35B-A3B LiteRT-LM Models

This repository is a conversion of the text decoder of Qwen/Qwen3.6-35B-A3B to LiteRT-LM, made with litert-torch export_hf (main 67f11e8) for LiteRT-LM built from source (main 02d7e8d8): on an Apple M4 Max (128 GB) the int8 GPU file decodes 76.99 tok/s and prefills 1,336.22 tok/s on the GPU (WebGPU, 2026-11-13), and the CPU file decodes 7.74 tok/s and prefills 101.45 tok/s with 12 threads (2026-11-12).

LiteRT-LM 0.19.2, the latest release on PyPI on 2026-11-13, does not run these files: its moe kernels accept only GELU, and SiLU is in two open LiteRT pull requests, #11192 and #11188; its GPU delegate also refuses the GPU files' odml.gated_delta_block (build the runtime).

Files

File Backend Quantization Context Size
qwen36_35b_a3b_int8_gpu_ekv4096.litertlm GPU linear layers int8 (per channel), the experts included; the router, the shared expert's gate and the token table fp32; asks for fp16 activations 4096 36.43 GB
qwen36_35b_a3b_mixed_int8_gpu_ekv4096.litertlm GPU experts int8 (per channel), the rest fp32; asks for fp32 activations 4096 42.15 GB
qwen36_35b_a3b_mixed_int8_cpu_ekv4096.litertlm CPU experts int8 (per channel), the rest fp32 4096 42.15 GB

The mixed int8 GPU file keeps every weight but the experts in fp32. Text in, text out: the vision encoder and the multi-token prediction layers are not in the files (conversion notes). The shared expert's gate (one output) stays float in every file.

Performance

File Backend Prefill (tok/s) Decode (tok/s) Peak footprint
qwen36_35b_a3b_int8_gpu_ekv4096.litertlm GPU (WebGPU) 1,336.22 76.99 49 GB
qwen36_35b_a3b_mixed_int8_gpu_ekv4096.litertlm GPU (WebGPU) 1,162.21 39.68 60 GB
qwen36_35b_a3b_mixed_int8_cpu_ekv4096.litertlm CPU (XNNPACK, 12 threads) 101.45 7.74 20 GB

Apple M4 Max (128 GB), 1,024 prefill and 256 decode tokens, measured on 2026-11-13 (int8 GPU file), 2026-11-13 and 2026-11-14 (mixed int8 GPU file) and 2026-11-12 (CPU) (how it was measured).

On the Apple M4 Max, the int8 GPU file decodes fastest of the three files (76.99 tok/s on WebGPU).

Accuracy

File Backend Same verdict as Transformers Ends with the stop token 3 turns
qwen36_35b_a3b_int8_gpu_ekv4096.litertlm GPU (WebGPU) 12 / 12 12 / 12 pass with filter_channel_content_from_kv_cache false
qwen36_35b_a3b_mixed_int8_gpu_ekv4096.litertlm GPU (WebGPU) 12 / 12 12 / 12 pass with filter_channel_content_from_kv_cache false
qwen36_35b_a3b_mixed_int8_cpu_ekv4096.litertlm CPU (XNNPACK, 12 threads) 12 / 12 12 / 12 pass with filter_channel_content_from_kv_cache false

12 questions and a 3-turn conversation with thinking on, greedy, judged against Transformers (bf16) (questions and method).

The int8 GPU file asks the runtime for fp16 activations, the mixed int8 GPU file for fp32. With fp32 activations the int8 GPU file passed the same checks but decoded 75.03 tok/s and prefilled 1,173.33 tok/s. With fp16 it also copied two 40-digit codes and recalled two numbers placed past position 2,048 of a 2,600-token prompt, as it did with fp32. The mixed int8 GPU file asking for fp16 left one of the 12 answers unfinished: its thought ran to the 2,048-token limit (GPU files).

q / k norm weights checked (2026-11-13): in each GPU file's 10 attention layers they equal 1 + the source's, with the GPU composites' partial-rotary permutation applied once, 20 of 20 in each of its 3 signatures (GPU files).

The CPU file runs in fp32 on the CPU: its only int8 weights are the experts', which the CPU moe kernel turns back into fp32 before it uses them (read in the build's source), so LiteRT-LM's run-time activation quantization of int8 linear layers (XNNPACK qd8) does not apply.

Run

./litert_lm_advanced_main --backend=gpu --model_path=qwen36_35b_a3b_int8_gpu_ekv4096.litertlm \
  --max_output_tokens=2048 --min_log_severity=2 --sampler_top_k=1 \
  --cache_dir=:nocache --sampler_handles_input=false --input_prompt="What is 17 + 25? Answer briefly."

./litert_lm_advanced_main --backend=cpu --model_path=qwen36_35b_a3b_mixed_int8_cpu_ekv4096.litertlm \
  --max_output_tokens=2048 --min_log_severity=2 --sampler_top_k=1 \
  --cache_dir=:nocache --num_cpu_threads=12 --input_prompt="What is 17 + 25? Answer briefly."

For the mixed int8 GPU file, put its name in the GPU command. Every GPU run of the CLI here (the 12 questions, the conversations, the benchmark) used --sampler_handles_input=false; the default (true) was not checked.

For a multi-turn chat with thinking on, set the Conversation's filter_channel_content_from_kv_cache to false (C++ ConversationConfig::Builder::SetFilterChannelContentFromKvCache(false)) or turn thinking off (SetThinkingConfig(ThinkingConfig(false, -1))); litert_lm_advanced_main has no flag for either, and the checks used a build of the CLI with two added flags that set them (conversations).

Limitations

  • One conversation per engine: a new conversation on the same engine does not reset the Gated DeltaNet state (LiteRT-LM issue #3165).
  • Once a message fails because it does not fit in the context window, every later message in that conversation fails as well on LiteRT-LM 0.18.1 and 0.19.1 (the next one with Task failed with state: 7; main 02d7e8d8 has the same code): start a new conversation after such an error, on a new engine as above.
  • Multi-turn with thinking on, in LiteRT-LM's default Conversation (filter_channel_content_from_kv_cache true): in the 3-turn check, the thought of turn 3 did not quote the user's turn-1 message. The CPU file and the mixed int8 GPU file took the number from the reply to turn 1, which repeats it; the int8 GPU file answered "I don’t know—it’s up to you to choose!". With the replies capped at 64 tokens (CPU file), so that the number is only in the user's message, the thought of turn 3 said it did not know the user's lucky number. With the setting false, the thought of turn 3 quoted the user's turn-1 message and turn 3 answered 37; with thinking off, turn 3 answered 37 (1 run each). In a second conversation with a list of 80 boxes in turn 1 (the int8 GPU file), the default's turn 3 said it did not have the context, and with the setting false it gave the right number (conversations).
  • Measured on a desktop only (Apple M4 Max): the files (36.43 to 42.15 GB) do not fit in a phone's memory, and the Android GPU, the Android CPU and iOS / iPhone were not measured.
  • The GPU program cache was not checked: the GPU runs behind the tables above used --cache_dir=:nocache.
  • Tool calls were not checked with these files (tool calls).

License

Apache License 2.0, the license of Qwen/Qwen3.6-35B-A3B.

Citation

@misc{qwen36_35b_a3b,
    title = {{Qwen3.6-35B-A3B}: Agentic Coding Power, Now Open to All},
    url = {https://qwen.ai/blog?id=qwen3.6-35b-a3b},
    author = {{Qwen Team}},
    month = {April},
    year = {2026}
}
Downloads last month
115
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for litert-community/Qwen3.6-35B-A3B

Quantized
(1,000)
this model