README.md

# HunyuanImage-3.0-Instruct-Distil · MXFP4 mixed-precision (`auto-round` / vLLM INC path)

This is the **`--format auto_round`**
the MXFP4-for-experts / MXFP8-for-everything-else recipe, exporting
`quant_method = "auto-round"` so vLLM loads it through the **INC** dispatch instead of the
`compressed-tensors` one. AR-only, DiT-only and AR+DiT were all run successfully on
vLLM 0.29.0 + vLLM-Omni `main`.

![AR+DiT full-pipeline output](images/ar_dit_full_tp1x1.png)

> Generated output, not a ground-truth reference: AR+DiT on **2 GPUs** (AR TP1 + DiT TP1),
> `seed=42`, 8 steps, prompt `A cute cat`.

---

## ⚠️ Read this first

1. **On Hopper (SM90: H100/H200) this is a memory saving, not a speed saving.** There are no FP4
   tensor cores, so the experts run through Marlin as **weight-only FP4 (effectively W4A16)**:

   ```
   Using MarlinExperts (weight-only FP4) for AutoRound MXFP4 MoE
   Using MarlinMxfp8LinearKernel for MXFP8 GEMM
   ```

   Blackwell (`b12x` / CUTLASS MXFP4) paths exist in vLLM but are **untested** for this build.

---

## Overview

| Field | Value |
|---|---|
| Base model | `tencent/HunyuanImage-3.0-Instruct-Distil` (`cfg_distilled=true`, `use_meanflow=true`) |
| MoE geometry | 32 layers × 64 routed experts, `moe_topk=8`, 1 shared expert/layer, hidden 4096, `moe_intermediate=3072` |
| Scheme | **Mixed**: routed experts → **MXFP4** (E2M1, group 32, E8M0); all other quantized `Linear` → **MXFP8** (E4M3, group 32, E8M0) |
| Export format | **`--format auto_round`** → `quant_method = "auto-round"`, `packing_format = "auto_round:llm_compressor"` |
| Loader path | vLLM **INC** (`INCConfig.override_quantization_method()` maps `"auto-round"` → `"inc"`) |
| Quantization tool | auto-round **0.15.0**, `--model_free` (no calibration data); wall time **707 s** on 1 GPU |
| Disk size | **52.59 GB** (48.98 GiB) — MXFP8-ct build 91.25 GB, BF16 base 158 GB ⇒ **0.33× BF16** |
| Parameters | total **83.04 B**: routed experts **77.31 B (93.1 %) @4-bit**, `self_attn` 1.49 B + `shared_mlp` 1.21 B @8-bit, **3.03 B BF16/FP32** |
| Expert tensor layout | `…experts.N.{gate_and_up_proj,down_proj}.weight_packed` = `uint8 [N, K/2]` (two E2M1 per byte, **low nibble = even column**) + `weight_scale` = `uint8 [N, K/32]` (E8M0: `value = 2^(byte−127)`) |
| Kept in BF16 / FP32 | ViT (`vision_model`, BF16), VAE (FP32, as in the base model), `wte`/`lm_head`, `guidance_emb`, `timestep_emb`, `timestep_r_emb`, `patch_embed`, `final_layer`, `time_embed*`, `vision_aligner`, and the **MoE router of all 32 blocks** (`…mlp.gate.wg`) |

`quantization_config` from `config.json` (verbatim, minus the long `extra_config`):

```json
{
  "quant_method": "auto-round",
  "packing_format": "auto_round:llm_compressor",
  "bits": 8, "group_size": 32, "sym": true, "data_type": "mx_fp",
  "act_bits": 8, "act_data_type": "mx_fp", "act_dynamic": true, "act_group_size": 32, "act_sym": true,
  "model_free": true, "iters": 0, "enable_quanted_input": false,
  "autoround_version": "0.15.0",
  "block_name_to_quantize": "model.layers",
  "extra_config": {
    ".*mlp\\.experts": { "bits": 4 },                       // ← the fix; stock export has ".mlp.experts."
    "final_layer.model.0.emb_layers.1": { "bits": 16, "data_type": "float", "act_bits": 16, "act_data_type": "float" },
    "... 220 more entries, all bits:16 overrides ..."
  }
}
```

Note there is **no `ignore` list** in this format (unlike the `compressed-tensors` sibling): the
16-bit exceptions are expressed as `extra_config` entries with `bits: 16`, and the non-Linear modules
that never quantize anyway (VAE convolutions, ViT patch/position embeddings) simply are not listed.

> `act_*` fields describe the *declared* scheme. What actually runs on SM90: dynamic MXFP8 activation
> quantization for dense `Linear`, but **no activation quantization for the experts** (Marlin W4A16).



---

## Quantization command

```bash
auto-round \
  --model_name tencent/HunyuanImage-3.0-Instruct-Distil \
  --model_free \
  --scheme MXFP8 \
  --layer_config '{.mlp.experts.:{bits:4,data_type:mx_fp}}' \
  --ignore_layers "vision,guidance_emb,timestep_emb,timestep_r_emb,final_layer,wte" \
  --format auto_round \
  --device cuda:0 \
  --output_dir ./HunyuanImage-3.0-Instruct-Distil-MXFP4-AutoRound

```

- Do **not** add a bare `gate` to `--ignore_layers`: it expands to `.*gate.*` and silently swallows the
  fused expert projections `gate_and_up_proj` (4096 tensors).
- The MoE router needs no entry: AutoRound hard-skips anything matching `.gate.`.
- `vision` must stay ignored: ViT `mlp.fc2` input dim 4304 is not divisible by the 32-element MX block.
- The run log echoes `Scheme: … bits=8` / `Packing format: mxfp8-quantized` — that is the **default**
  scheme only, not evidence that `--layer_config` was ignored. Check the artifact instead
  (`weight_packed` count must be 4096).

---

## Inference environment

| Component | Version |
|---|---|
| vLLM | **0.29.0** |
| vLLM-Omni | latest `main` (validated at `1c7476ec`, `0.29.0rc2.dev161+g1c7476ec1`, editable) |
| PyTorch | 2.13.0+cu132 |
| FlashInfer | 0.6.18 |
| GPU | NVIDIA H200 141 GB — AR-only 1×, DiT-only 2×, AR+DiT 2× |

```bash
pip install vllm==0.29.0 && pip install -e /path/to/vllm-omni
python -c "import vllm, vllm_omni, os; print(vllm.__version__, os.path.dirname(vllm_omni.__file__))"

# host-specific, drop if unneeded here
export NCCL_NVLS_ENABLE=0
export VLLM_USE_FLASHINFER_SAMPLER=0
cd /tmp            # cwd must be neutral: children re-import vllm_omni from cwd
```

---

## Validated run modes (2 GPUs, cards 0 and 1)

| Mode | deploy YAML | Result | Evidence from logs |
|---|---|---|---|
| AR-only | `hunyuan_image3_ar_tp1.yaml` (TP1) | ✅ | `Model loading took 46.69 GiB`; MXFP4 MoE dispatch line above; text `It's a warm and healing style` (matches the `-mixed` and MXFP8 builds verbatim) |
| DiT-only | `hunyuan_image3_dit_tp2.yaml` (TP2) | ✅ image | `Model loading took 25.3899 GiB`/card; `Saved generated image …` (std 49.34) |
| AR+DiT | `hunyuan_image_3_moe_2gpu_tp1.yaml` (TP1+TP1) | ✅ image | both stages loaded (46.69 + 46.63 GiB); `Saved generated image …` (std 46.44) |

Before the metadata fix, **all three** failed with the `KeyError` quoted at the top.

```bash
export CUDA_VISIBLE_DEVICES=0,1
cd /tmp
python /path/to/vllm-omni/examples/offline_inference/text_to_image/text_to_image.py \
  --model               /path/to/HunyuanImage-3.0-Instruct-Distil-MXFP4-ar \
  --deploy-config       ./hunyuan_image_3_moe_2gpu_tp1.yaml \
  --prompt              "A cute cat" \
  --num-inference-steps 8 \
  --guidance-scale      5.0 \
  --seed                42 \
  --output              ./output.png
```

The two-stage YAML (`stage 0` AR with `omni_kv_config.need_send_cache`, `stage 1` DiT with
`need_recv_cache`, joined by a `SharedMemoryConnector`) is reproduced in the sibling build's card;
`devices` are **local** indices, combine with `CUDA_VISIBLE_DEVICES`.

**Parameter notes** — `--num-inference-steps 8` (distilled: do not use 50); `--guidance-scale` is a real
input (`cfg_distilled=true`, vLLM-Omni feeds `1000 × guidance_scale` as the guidance embedding; the image
above used `5.0`, vLLM-Omni's Distil e2e reference uses `2.5`); `--prompt` reaches only the AR stage.

---

## Fidelity

### Weight side — identical to the sibling build by construction

The tensors are byte-identical, so the dequantization error is the same:

| Module | scheme | rel. L2 vs BF16 | ≈ effective bits |
|---|---|---:|---:|
| routed experts `gate_and_up_proj` / `down_proj` | MXFP4 | **0.1120 – 0.1124** | 3.65 |
| `self_attn.*`, `shared_mlp.*` | MXFP8 | 0.0267 | 5.73 |
| router / ViT / VAE / embed (unquantized) | BF16/FP32 | 0.0000 | — |

MXFP4 experts carry **4.21×** the relative error of the MXFP8 group (**17.7×** in noise power) on
**93 %** of the parameters, and this equals the ideal per-group E8M0 search (optimized RTN) to four
decimals — i.e. it is the format's own floor, not a tuning miss.

### Image side (DiT-only TP2 — topology matched, noise floor 41.4 dB)

| Candidate | vs BF16 PSNR | SSIM@1 | @1/2 | @1/4 | **@1/8** | std |
|---|---:|---:|---:|---:|---:|---:|
| MXFP8-ct (for reference) | 31.59 | 0.979 | 0.976 | 0.978 | 0.982 | 53.62 |
| **this build (INC path)** | **12.91** | **0.6698** | 0.5417 | 0.3910 | **0.2611** | 49.34 |
| sibling `-mixed` (CT path) | 12.99 | 0.6720 | 0.5449 | 0.3963 | 0.2707 | 48.86 |
| **this build vs `-mixed`** | **33.03** | 0.9799 | 0.9772 | 0.9758 | **0.9822 (flat)** | — |

| BF16 base (DiT TP2) | this build (DiT TP2) | sibling `-mixed` (DiT TP2) |
|---|---|---|
| ![bf16](images/bf16_dit_only_reference.png) | ![ar](images/dit_only_tp2.png) | ![mix](images/sibling_llm_compressor_dit_only.png) |

- SSIM **falls with coarser scale** vs BF16 ⇒ **composition-level divergence**, not a texture/style
  shift. The image is a valid, un-degraded *different* picture of a cute cat.
- This build vs the sibling is **flat across scales** ⇒ same composition, small texture-level numeric
  differences from the different kernel entry points.

### Side-by-side montages

**Figure A — DiT-only, TP2 for all four builds (topology matched ⇒ the readable axis).**
③ and ④ additionally share the *loader format* (`auto-round`/INC), so **③↔④ isolates the experts bit
width as the only variable.** Each cell carries its topology and PSNR / SSIM@1 / **SSIM@1/8** vs BF16
(SSIM on 1/8-scale images keeps only low-frequency structure ⇒ "same composition or not").

![DiT-only: BF16 / MXFP8-ct / MXFP8-ar / this build](images/grid_ar_ditonly.png)

| comparison | what differs | PSNR | SSIM@1 | **SSIM@1/8** |
|---|---|---:|---:|---:|
| ② ↔ ③ (`MXFP8-ct` vs `MXFP8-ar`) | **only the export format** | **44.46** | 0.9965 | **0.9984** |
| ③ ↔ ④ (`MXFP8-ar` vs this build) | **only experts 8→4 bit** | 12.71 | 0.6623 | **0.2518** |
| ② ↔ `-MXFP4-mixed` (CT side, for cross-check) | only experts 8→4 bit | 12.81 | 0.6643 | **0.2617** |
| ① ↔ ② vs BF16 | MXFP8 | 31.59 | 0.9788 | 0.9823 |
| ① ↔ ③ vs BF16 | MXFP8 (INC) | 31.79 | 0.9811 | 0.9842 |
| ① ↔ ④ vs BF16 | MXFP4 (INC) | 12.91 | 0.6698 | 0.2611 |

Noise floor: re-running the *same* build with the *same* command gives **41.4 dB / SSIM 0.995**.

⇒ Three things worth reading off this figure:
1. **Two MXFP8 formats agree almost perfectly (44.5 dB, above the re-run floor)** ⇒ export format is
   immaterial at MXFP8.
2. **Dropping the experts to MXFP4 moves SSIM@1/8 from ~0.98 to ~0.25 — and both independent pairings
   (INC side 0.2518, CT side 0.2617) agree.** That is a *composition-level* change, not a style/texture
   shift, and it is a property of MXFP4, not of either loader.
3. Unlike SSIM-vs-BF16, the ③↔④ pair holds topology *and* format fixed, so the 0.25 cannot be blamed on
   either confounder.

**Figure B — AR+DiT full pipeline (only ③↔④ are topology-matched).**

![AR+DiT: BF16 / MXFP8-ct / MXFP8-ar / this build](images/grid_ar_ardit.png)

| comparison | topology | PSNR | SSIM@1/8 |
|---|---|---:|---:|
| ① ↔ ③ (`BF16` 4-GPU vs `MXFP8-ar` 2-GPU) | **cross-topology** | 26.85 | 0.8715 |
| ③ ↔ ④ (`MXFP8-ar` vs this build) | **both 2-GPU TP1+TP1** | 12.88 | **0.1889** |

⚠ The full pipeline **re-draws the AR CoT every run** — the AR text + ratio tokens are the DiT's
conditioning via `omni_kv_config` (same-build AR+DiT re-run: **27.8 dB**; and two builds with
*byte-identical* weights still disagree at SSIM@1/8 = 0.23 in this mode while agreeing at 0.98 in
DiT-only mode). Cross-topology alone already costs 0.87 → 0.98. **Read Figure A, not Figure B, for
quantization fidelity; Figure B only shows that this build renders a sane image.**

### Not measured

No task-level benchmark (GenEval / DPG-Bench / CVTG-2K / DrawBench) has been run for this build or for
the `-mixed` sibling. "Different composition" is measured; "worse quality" is not established.

---

## Known limitations

1. **Requires the metadata fix** above; without it the checkpoint does not load at all (fails loudly,
   which is preferable to loading silently with the wrong scheme).
2. **Memory-only on Hopper** — experts run W4A16 through Marlin.
3. **Composition diverges from BF16/MXFP8** (SSIM@1/8 ≈ 0.26); do not assume parity, and do not mix
   outputs from this build with outputs from the `-mixed` build in one A/B comparison (33 dB apart).
4. **The ViT stays BF16** (4304 % 32 ≠ 0) ⇒ not a whole-model MXFP4 build.
5. The upstream INC matcher's MoE branch is keyed on a class name containing `fusedmoe`, which never
   matches `RoutedExperts` on vLLM ≥ 0.29; this build works around the resulting brittleness by using an
   explicit regex key. A cleaner fix belongs in AutoRound (emit a matchable key) or in the INC parser.
6. `block_name_to_quantize = "model.layers"` scopes quantization to the 32 shared backbone layers; AR and
   DiT load the same tensors through different stacks.
7. `images/` holds verification snapshots only; it is not read when loading weights.