vmlinux/Artemis-31B-v1.2-Q4_K_M-NInfer

Artemis 31B v1.2 Q4_K_M for NInfer

A NInfer conversion of TheDrummer's Artemis 31B v1.2, based on Bartowski's Q4_K_M GGUF. The conversion preserves the existing quantization.

Built for a single NVIDIA Tesla V100 with 32 GB VRAM (sm_70, CUDA 12.8). I run the JimmyMax/ninfer-v100 fork at revision f4aab2ae632afa32857bb15c32eafb29851cea47 with the included Artemis compatibility patch.

I've used this build extensively for creative writing, and I love the results.

Download artemis-Q4_K_M-text.ninfer (19.53 GB). My Artemis settings:

ninfer-serve ./artemis-Q4_K_M-text.ninfer \
  --max-context 262144 --max-concurrency 1 \
  --kv-dtype int8 --prefill-chunk 2048 \
  --host-kv-mib 512 --ngram-draft-tokens 0

MTP is off for this build.

Thanks to TheDrummer for the model, Bartowski for the Q4_K_M quant, and JimmyMax and the NInfer contributors for the engine.