bluemorpholimited/Morpho-72B-MoE

One dense giant. Sixty-four specialist minds. Morpho-180B takes the battle-tested open dense foundation and grows it into a routed collective: every token is served by the two domain experts that know it best, plus a shared generalist that never sleeps. Train each expert to convergence. Leave none behind.

---

⚡ Why Morpho

• 🧠 180B total / ~78B active per request — giant knowledge, agile inference.
• 🎯 64 domain experts, zero overlap — every training example maps to exactly one expert. Auditable by construction.
• 📜 Fully auditable data pipeline — raw → tagged → tokenized corpora, code included, in Morpho-72B-MoE-Data.
• 🔁 Resume-first engineering — every expert checkpoint (weights + loss + step) is versioned in Morpho-72B-MoE-Checkpoints. Any box can pick up exactly where the last one died.
• 🌌 128K context via RoPE YaRN for long documents, codebases, and books.

---

🏗 Architecture

Component	Spec
Base	72B dense backbone, 4-bit NF4 QLoRA (frozen)
Layers	80 (40 dense even + 40 MoE odd)
Hidden	8192 · 64 Q heads / 8 KV heads (GQA) · SwiGLU · RMSNorm
Router	Learned, top-2 of 8 per MoE layer + 1 shared expert
Adapters	LoRA rank-128 / alpha-256 on q,k,v,o,gate,up,down (1.68B trainable params per expert, 1,120 tensors)
Context	128K via RoPE YaRN
Total / active	~180B / ~78B per request

What ships in this repo: the merged dense 72B backbone with all 64 expert deltas folded in, plus the GGUF builds. The 180B figure describes the full collective — 64 converged adapters that each specialise a 72B base. The merged model is the convenient default; the experts remain available as adapters.

---

🗺 The 64 experts

Eight sectors — Finance, Law, Medicine, Technology, Science, Business, Arts, Humanities — each with 8 specialists. Full roster with per-expert data counts is logged at every training run; counts range from 606 to 92,366 examples per expert (median 6,334), 822,745 total, each mapped to exactly one expert.

♻️ Data refinery — every byte auditable

🛠 Build pipeline

flowchart LR
    A["1 · Base<br/>72B dense"] --> B["2 · Data<br/>822,745 routed"]
    B --> C["3 · MoE structure<br/>64 routed experts"]
    C --> D["4 · SFT<br/>64/64 converged"]
    D --> E["5 · DPO<br/>31/31 converged"]
    E --> F["6 · Merge<br/>gate PASS · PPL ×1.02"]
    F --> G["7 · Verify<br/>GSM8K 76.7% · PPL 7.68"]
    G --> H["8 · GGUF<br/>f16 · Q8_0 · Q5_K_M · Q4_K_M"]
    H --> I["9 · Distill<br/>7B + 1.5B students"]
    style F fill:#39ff88,stroke:#00f0ff,color:#000
    style I fill:#39ff88,stroke:#00f0ff,color:#000

🗺 Roadmap status — what's done, what's moving

Stage	Status	Detail
1 · Base (72B dense, 4-bit QLoRA)	✅ done	frozen backbone, all experts train on it
2 · Data (822,745 ex, v5 taxonomy, code+manifests)	✅ done	auditable in Morpho-72B-MoE-Data
3 · MoE structure (64 routed LoRA, top-2+shared, 128K YaRN)	✅ done	design locked, see Architecture
4 · SFT (per-expert to convergence)	✅ done — 64/64	all experts converged to full patience
5 · DPO (preference alignment)	✅ complete	31/31 experts trained to convergence on Morpho-72B-MoE-DPO (best losses 3.9e-08 → 2.7e-02); 33 thin experts (<200 pairs) keep SFT-only by policy
6 · Merge (unified MoE)	✅ done — gate PASS	64 converged adapters (DPO where available, SFT otherwise) → one bf16 72B model via sign-consensus (TIES-style) merge. PPL 7.69 → 7.83 (×1.02), text fluent, on-topic in all 8 domains. Published as Morpho-72B-MoE-Merged (45 files, 145.4 GB). Gate history in Stage 6 below
7 · Verify (benchmarks + audit)	✅ done	GSM8K 76.7% (23/30) vs base 83.3% (25/30); PPL 7.68 vs 7.56; 19.5 tok/s. Full table in Stage 7
8 · GGUF (quantized release)	✅ done	f16 (4 shards) + Q8_0 + Q5_K_M + Q4_K_M, 324.5 GB, in Morpho-72B-MoE-GGUF
9 · Distill (compact students)	✅ done	Qwen2-7B → Morpho-7B-Distill (best loss 0.7188, PPL 11.32, 15.2 GB) · Qwen2-1.5B → Morpho-1.5B-Distill (best loss 1.5750, PPL 13.16, 3.1 GB). Details in Stage 9

Right now: the pipeline is 9/9 complete. Merged weights, four GGUF formats, and both distilled students are published. Every stage passed its own gate before anything was published — no unvalidated weights ever left the box.

📉 Convergence gallery — every expert trains to its floor

🚦 Stage 6 — the merge gate

We do not publish a merge because it finished. We publish it because it passed a perplexity gate against the base model on the same 40-document holdout, with a human read of the generations behind the number.

Merge variant	Cap	Base PPL	Merged PPL	Ratio	Verdict
Plain delta mean	—	7.69	300.88	39.1×	❌ rejected — degenerate text
Sign consensus	0.020	7.69	49.74	6.47×	❌ rejected
Sign consensus	0.011	7.69	9.34	1.21×	❌ rejected — passed the early gate, but output had broken grammar
Sign consensus	0.006	7.69	7.83	1.02×	✅ shipped

Method (TIES-style): per tensor, trim the smallest 80% of delta magnitude, elect a sign per coordinate by weighted vote, sum only consenting deltas, then per-module relative-norm cap so no single expert dominates a layer. δ = B @ A · (alpha/r = 2), 560/560 tensors merged in 15.1 min. Mean |δ|/|W| was 2.70% before the cap and 0.6% after it; the largest single relative change was 19.3%.

The 1.21× candidate is why the gate is 1.10×: it looked acceptable statistically and still produced ungrammatical text. Numbers alone are not a release criterion here.

📊 Stage 7 — verification

Same harness, same prompts, 4-bit NF4 for both models, greedy decoding.

Run	GSM8K	Perplexity (40 docs)	Throughput
Base 72B dense	25/30 · 83.3%	7.56	17.1 tok/s
Morpho-72B-MoE-Merged	23/30 · 76.7%	7.68	19.5 tok/s
Δ	−2 items (~1 σ at n=30)	+1.6%	+14%

Domain audit: 8/8 sector prompts (Finance, Law, Medicine, Engineering, Science, Math, Code, Writing) produced coherent, on-topic answers. The merged model is quality-neutral against its base — that is the honest claim. It is not a benchmark gain; the specialists live in the adapter repos, and an additive soup of 64 domain deltas cannot preserve 64 specializations at once.

📦 Stage 8 — GGUF builds

Artifact	Format	Size
f16/Morpho-180B-f16-0000{1..4}-of-00004.gguf	f16 (4 shards)	145.4 GB
Morpho-180B-q8_0.gguf	Q8_0	77.3 GB
Morpho-180B-q5_k_m.gguf	Q5_K_M	54.4 GB
Morpho-180B-q4_k_m.gguf	Q4_K_M	47.4 GB
Total		324.5 GB

Built with llama.cpp convert_hf_to_gguf.py + llama-quantize, published to Morpho-72B-MoE-GGUF. Point Ollama or llama.cpp at the Q4_K_M file for a single-GPU workstation.

🎓 Stage 9 — distilled students

Teacher: the merged 72B, loaded in 4-bit NF4. Students: full fine-tune. Objective: L = α·T²·KL(s/T ‖ t/T) + (1−α)·CE, shifted causal, T = 2, α = 0.5, 2,500 steps, 5 rotating checkpoints, best-by-loss kept.

Two details that matter more than the hyperparameters:

• The corpus is re-rendered through the student's chat template. Our first run trained on raw Instruction: … / Response: … rows, and the students dutifully learned to emit Response: at the start of every answer. Fixed by rebuilding the corpus as real chat turns.
• Loss is masked to the answer span. Prompt tokens contribute no gradient, so capacity goes into answering, not into echoing questions.

Student	Repo	Best loss	GSM8K	PPL (10 docs)	Loop-free domains
Qwen2-7B	Morpho-7B-Distill · 15.2 GB	0.7188	1/10	11.32	6/8 greedy → 8/8 at repetition_penalty=1.1
Qwen2-1.5B	Morpho-1.5B-Distill · 3.1 GB	1.5750	2/10	13.16	2/8 greedy → 8/8 at repetition_penalty=1.1

Read these for what they are. The students are compact approximations of the teacher's writing distribution, not replacements for it:

• They inherit fluency, tone, and domain vocabulary — every one of the eight sector prompts gets a relevant, well-formed answer with the recommended decoding settings.
• They do not inherit the teacher's arithmetic reliability. The 7B student scores 1/10 on GSM8K; use the 72B for anything where the number matters.
• Small students loop under pure greedy decoding. repetition_penalty=1.1 removes every observed loop in our 8-domain audit, and it is the setting we recommend in the snippet below.

out = model.generate(**ids, max_new_tokens=256, do_sample=False, repetition_penalty=1.1)

Honest summary: the 7B student is a usable fast drafter and a decent style transfer of the merged teacher; the 1.5B is a demo-sized model that shows the distillation pipeline works end to end, not a knowledge store.

🧩 Where the specialists live

The 64 domain experts are not thrown away by the merge. Each converged adapter (64 SFT, 31 with a DPO refresh) is versioned in Morpho-72B-MoE-Checkpoints and Morpho-72B-MoE-DPO. Load the merged model for general work; load base + a single adapter when you want one domain at full specialisation. That is the intended division of labour.

⚙️ Quick start

# merged model (transformers)
pip install -U transformers accelerate
python -c "
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained('bluemorpholimited/Morpho-72B-MoE-Merged', device_map='auto', torch_dtype='bfloat16')
t = AutoTokenizer.from_pretrained('bluemorpholimited/Morpho-72B-MoE-Merged')
print(t.decode(m.generate(**t('Explain stare decisis.', return_tensors='pt').to(m.device), max_new_tokens=200)[0], skip_special_tokens=True))
"

# quantized single-GPU path
huggingface-cli download bluemorpholimited/Morpho-72B-MoE-GGUF Morpho-180B-q4_k_m.gguf --local-dir .
llama-cli -m Morpho-180B-q4_k_m.gguf -p "Explain stare decisis." -n 256

📚 Citation

@misc{morpho180b,
  title  = {Morpho-180B: a 64-expert domain collective with a gated merge pipeline},
  author = {Blue Morpho Limited},
  year   = {2026},
  note   = {Merged 72B dense release, GGUF quantizations, and distilled students},
  url    = {https://huggingface.co/bluemorpholimited/Morpho-72B-MoE}
}

Apache 2.0, inheriting the Qwen2-72B license terms. Every number on this card was produced by the scripts in the data repo — nothing is estimated.