steampunque/Deepseek-R1-Distill-Llama-8B-Hybrid-GGUF

GGUF hybrid layer quantization of Deepseek R1 Distill Llama 3.1 8B Instruct by deepseek-ai

Original model: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. This particular quant achieves a ~6.5G gguf with improved performance compare to a ~6.6G Q6_K GGUF. The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. For this file the layer quants are as follows:

Q5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k
Q6_K_S : Q6_K
Q6_K_M : attn_v = q8_0 ffn_d = q8_0
Q6_K_L : attn_v = q8_0 attn_o = q8_0 ffn_d = q8_0

   LAYER_TYPES='[
   [0 ,"Q6_K_S"],[1 ,"Q5_K_L"],[2 ,"Q5_K_M"],[3 ,"Q5_K_M"],[4 ,"Q5_K_M"],[5 ,"Q5_K_M"],[6 ,"Q5_K_M"],[7 ,"Q5_K_M"],
   [8 ,"Q5_K_L"],[9 ,"Q5_K_M"],[10,"Q5_K_L"],[11,"Q5_K_M"],[12,"Q5_K_L"],[13,"Q5_K_M"],[14,"Q5_K_L"],[15,"Q5_K_M"],
   [16,"Q5_K_L"],[17,"Q6_K_S"],[18,"Q5_K_L"],[19,"Q6_K_S"],[20,"Q5_K_L"],[21,"Q6_K_S"],[22,"Q5_K_L"],[23,"Q6_K_S"],
   [24,"Q6_K_S"],[25,"Q6_K_M"],[26,"Q6_K_M"],[27,"Q6_K_M"],[28,"Q6_K_L"],[29,"Q6_K_L"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The quants was updated on 12/20/2025 optimized for best performance across a small set of curated test prompts. The new quant hits 100% accuracy on BBH object counting eval for both text-only and audio (ultravox) prompting.

Comparison:

Quant	size	PPL	Comment
Q6_K	6.6e9	13.1	Q6_K with default embedding and output
Q6_K_H	6.5e9	13.1	Hybrid quant with Q6_K embedding Q6_K output

Usage:

This model may be used together with fixie-ai ultravox-v0_5-llama-3_1-8b to enable it to process audio (.mp3 and .wav files) and text inputs and generate text outputs. The mmproj file is made available here: https://huggingface.co/steampunque/ultravox-v0_5-llama-3_1-8b-Hybrid-GGUF More information about running multimedia may be found in the docs in the mtmd readme in the tools directory of the llama.cpp source tree https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/README.md.

The model can be speculated using Qwen3 0.6B. Approximate performance on a 4070 with context and weights in VRAM using a custom downstream greedy speculator with fixed spec block length ND and dynamic vocab translation :

Prompt	ND	Gen TPS	Comment
goldcoin	0	68	non code
goldcoin	3	85	non code
humaneval	0	68	code
humaneval	4	103	code
humaneval	4	132	Spec with Qwen 2.5 coder 0.5B, disable think mode (pure code gen)

goldcoin:

I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equally with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river? Use step-by-step reasoning to solve this problem.

humaneval:

generate python code for the described function header:

from typing import List

def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """

Benchmarks:

A full set of benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
Deepseek-R1-Distill-Llama-8B-Instruct.Q6_K_H.gguf	Q6_K_H	6.5e9 B	~Q6_K size, higher performance
ultravox-v0_5-llama-3_1-8b.mmproj.gguf	F16	1.38e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040