Mac patch is missing vision projector code (FIXED using mtmd hunks from Spark patch)

#4
by nazeshinjite - opened

Hi, thanks for the Mac patch and the GGUFs! This is a really impressive model in my testing, and the quants you provided are perfect.

The README's Mac section builds 2cbca389f + hy4-flash-mac-minimal.patch, but that patch has no tools/mtmd changes. The hy_v4_flash projector type is unknown to that build, so the mmproj doesn't load and the llama-mtmd-cli image example fails on the Mac path (text works):

mtmd_get_memory_usage: error: Failed to load CLIP model from mmproj-Hy4-flash-bf16.gguf
srv    load_model: [mtmd] failed to get memory usage of mmproj

The tools/mtmd hunks of hy4-flash-spark.patch apply cleanly on top of 2cbca389f + the Mac patch (6 files: CMakeLists.txt, clip-impl.h, clip.cpp, models/hy-v4-flash.cpp, models/models.h, mtmd.cpp):

awk '/^diff --git/{keep=($3 ~ /^a\/tools\/mtmd\//)} keep' hy4-flash-spark.patch > hy4-flash-mtmd.patch
git apply hy4-flash-mtmd.patch

With them, mmproj-Hy4-flash-bf16.gguf loads on an M5 Max 128 GB (441 tensors, about 1036 MiB on Metal). Folding these hunks into the Mac patch, or adding the step to the README, would get vision working on Mac.

Patch files used: hy4-flash-mac-minimal.patch sha256 97f1213f..., hy4-flash-spark.patch sha256 6ec50722....

I am opening a PR to this repo with the updated patch, if you'd like to use it!

Sign up or log in to comment