wesjos/Ling-3.0-tiny-CuteHuman-GGUF

Ling-3.0-tiny · CuteHuman (GGUF)

The emoji-first chat persona, quantized for llama.cpp — Q4_K_M at 4.49 GiB, ready to serve.

_{Q4_K_M 4.49 GiB  ·  7.89 B total / 1.3 B active  ·  131,072 context  ·  text only  ·  MIT}

---

💾 Want the full-precision weights? → Ling-3.0-tiny · CuteHuman

The identical fine-tune in bf16: 14.70 GiB in 4 shards, load it with a plain from_pretrained — no adapter to mount. This repo is the quantized version for llama.cpp.

🖼 Demo

Both shots come from a chat client with the CoT trace collapsed — the WORK row is the reasoning panel, the ANSWER row is the reply. Turn it on to see the thinking trace.

中文 · 你好吗 → 计算8.2×9.12/2.3 → 描述鸵鸟的外观

Greeting, then a two-step arithmetic walk-through that keeps its decimals (74.784 ÷ 2.3 ≈ 32.51), then an ostrich description. Emoji come out unprompted.

_{中文 · “你好呀！😊 很高兴见到你！” → 分步算式 8.2 × 9.12 = 74.784 → 74.784 ÷ 2.3 ≈ 32.51 → 鸵鸟外观描述}

English · hey → nothing → im feel so sleepy in the morning → how is your day?

Small talk that stays warm, then sleep advice, then an answer about its own day.

_{English · “Heyy! What can I help you with today? 🌟” → “That’s cool! No worries at all. 😄” → morning advice → “My day? I’m always ready to listen and chat with you!”}

---

📦 Files

File	Size	What it is
ling3-cutehuman-v5-Q4_K_M.gguf	4.49 GiB	the model — 7.89 B params in 526 tensors (bailingmoe3 arch)

The GGUF carries its own chat template (tokenizer.chat_template, 6,041 B) — it is byte-identical to the upstream chat_template.jinja and to this model's own copy, so no --chat-template-file is needed.

---

🚀 Quickstart

llama-server -m ling3-cutehuman-v5-Q4_K_M.gguf \
  --jinja -c 32768 -ngl 99 --host 0.0.0.0 --port 18080 \
  --temp 1.0 --top-p 0.95 --top-k 20 --repeat-penalty 1.05

Then talk to it like any OpenAI endpoint:

curl -s http://127.0.0.1:18080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
        "messages": [{"role": "user", "content": "今天上班好累啊"}],
        "temperature": 1.0, "top_p": 0.95, "top_k": 20, "max_tokens": 1024,
        "chat_template_kwargs": {"enable_thinking": true}
      }'

llama-cli, one shot:

llama-cli -m ling3-cutehuman-v5-Q4_K_M.gguf --jinja -p "今天上班好累啊" \
  -n 256 -c 4096 --temp 1.0 --top-p 0.95 --top-k 20 --repeat-penalty 1.05

Default sampling

temperature=1.0   top_p=0.95   top_k=20
repeat_penalty (repetition_penalty)=1.05     # required
max_tokens 256~1024 · stop=["<|role_end|>"]  # required
enable_thinking=True (--jinja runs the upstream template)

---

💡 Notes

• The persona lives in the weights. The embedded chat template is byte-identical to upstream's — no persona is baked into the template — so an empty system prompt reproduces the screenshots.
• Emoji and kaomoji are the default register. A system prompt like 「少用表情，正常说话」 / "tone the emoji down" is enough to soften it.
• Thinking is on by default and the template opens a <think> block. For chit-chat send "chat_template_kwargs": {"enable_thinking": false}; otherwise the reasoning trace eats the token budget. This checkpoint was trained with CoT in the loss, so enable_thinking=true is the in-distribution setting.
• Context is 131,072 in the metadata (bailingmoe3.context_length), but the KV cache is what actually limits you — 32K is a realistic start on a 12–16 GB card.
• Needs a bailingmoe3-aware build. Native support landed in llama.cpp #26608 (merged 2026-08-17, build b10470); a dedicated Ling 3.0 chat parser followed in #28682 (merged 2026-09-19). Older builds report an unknown architecture.
• Converted from the merged bf16 checkpoint of wesjos/Ling-3.0-tiny-CuteHuman — go there for the full-precision weights and the transformers recipe.

---

⚠️ Limitations

• Q4_K_M is a 4-bit quant — expect a small quality drop against the bf16 original, most visible on long reasoning.
• Weak EOS prior (a base-model trait, not introduced by the tune). The model often does not emit <|role_end|> after answering and keeps going with next-question-style text. This happens on the base model too. The tokenizer's eos_token is <|role_end|>, so stop=["<|role_end|>"] is a backstop for the case where the turn is left open. → Always set stop=["<|role_end|>"] and repeat_penalty=1.05; DRY sampling helps further.
• 7.9 B total but only 1.3 B active per token — cheap to run, but long-chain reasoning is still weaker than a same-size dense model.
• A persona tune, not a safety tune — review outputs before putting it in front of users.
• The screenshots were taken under a chat client that is not shipped here. This repo has no webui/ directory, and the client in the shots shows WORK / ANSWER / STEPS / cached labels.

---

📁 Related

Repo	What it is
wesjos/Ling-3.0-tiny-CuteHuman	the bf16 full model — 14.70 GiB, 4 shards, from_pretrained
inclusionAI/Ling-3.0-tiny	the base model (MIT)
wesjos/Qwen3.5-9B-CuteHuman-GGUF	the same persona on a 9B dense model, quantized
wesjos/Spark-x2.6-4B-CuteHuman-GGUF	the same persona on a 4B model, quantized

_{Demo screenshots live in demos/.}

---

📄 Credits

• Base model: inclusionAI/Ling-3.0-tiny (MIT)
• Fine-tune: wesjos/Ling-3.0-tiny-CuteHuman