Instructions to use QuantTrio/Kimi-K3-Cubic-2.5Bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="QuantTrio/Kimi-K3-Cubic-2.5Bit", trust_remote_code=True)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("QuantTrio/Kimi-K3-Cubic-2.5Bit", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantTrio/Kimi-K3-Cubic-2.5Bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Kimi-K3-Cubic-2.5Bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/QuantTrio/Kimi-K3-Cubic-2.5Bit

SGLang

How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuantTrio/Kimi-K3-Cubic-2.5Bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Kimi-K3-Cubic-2.5Bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuantTrio/Kimi-K3-Cubic-2.5Bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Kimi-K3-Cubic-2.5Bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with Docker Model Runner:
```
docker model run hf.co/QuantTrio/Kimi-K3-Cubic-2.5Bit
```

benchmark data for Kimi-K3-Cubic-2.5Bit

by bencswong2 - opened 12 days ago

Discussion

bencswong2

12 days ago

Dataset: GPQA Diamond (198 questions)
Config: temperature 1.0, top_p 0.95, reasoning_effort max
Result: 178/198 = 89.9%

bencswong2

4 days ago

On the deepswe test set, when running with FP8_Q16, only 2 out of the first 11 questions were answered correctly. With BF16, this improved to 6 questions. I also found that running the model for an extended period (running deepswe) causes the model to malfunction — it keeps outputting '@@@' continuously, and GPU 0 draws around 400W while the other GPUs are at roughly 200-something W. In the end, using BF16, it ran through over 90 questions with an accuracy of about 51%. Based on extrapolation, the final score should land around 52–57%, which is far too far from the score of the full-version Kimi K3 model. I hope your next quantized model will come with better tuning.

tclf90

QuantTrio org 4 days ago

Thanks for testing! I’ve found quite a few floating-point bugs in vLLM that hurt model quality. Without careful fixes, even the BF16 model performs worse than llama.cpp, which avoids distributed inference and all-reduce.

Fixing this comes first, so quantization is a lower priority for now. Cubic quant has also been further optimized, and more BPW-efficient versions are coming. Stay tuned!

bencswong2 changed discussion status to closed 3 days ago

bencswong2 changed discussion status to open 3 days ago

bencswong2

3 days ago

Thanks for testing! I’ve found quite a few floating-point bugs in vLLM that hurt model quality. Without careful fixes, even the BF16 model performs worse than llama.cpp, which avoids distributed inference and all-reduce.

Fixing this comes first, so quantization is a lower priority for now. Cubic quant has also been further optimized, and more BPW-efficient versions are coming. Stay tuned!

Hi tclf90, thanks for the update and for prioritizing those floating-point bug fixes — that definitely makes sense as a first step.

In the meantime, I'd suggest using DeepSwe for testing, as it provides a more standardized and reproducible evaluation pipeline. I actually have a complete test script ready to go for this. If you're interested, just let me know and I can share it with you. It might help streamline the validation process once the BF16 fixes are in place, and could also be useful for benchmarking the upcoming Cubic quant versions.

Looking forward to the improvements — keep up the great work!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment