Apertus-8B-Instruct-2509-FP8

This is a premium FP8 quantized version of swiss-ai/Apertus-8B-Instruct-2509 for efficient inference.

Model Description

Property	Value
Base Model	Apertus-8B-Instruct-2509
Quantization	FP8 (E4M3 format) via llm-compressor
Target Hardware	NVIDIA Ada Lovelace & Hopper GPUs
Quantization Time	58.2 minutes
Calibration Samples	2,048 (premium multi-dataset)

Usage

With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "TevunahAi/Apertus-8B-Instruct-2509-FP8",
    torch_dtype=torch.float8_e4m3fn,  # FP8 dtype
    device_map="auto",
    low_cpu_mem_usage=True,
)

tokenizer = AutoTokenizer.from_pretrained("TevunahAi/Apertus-8B-Instruct-2509-FP8")

# Generate
messages = [{"role": "user", "content": "Explain quantum computing"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

With vLLM (Recommended for production)

from vllm import LLM, SamplingParams

llm = LLM(model="TevunahAi/Apertus-8B-Instruct-2509-FP8")
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)

prompts = ["Explain quantum computing"]
outputs = llm.generate(prompts, sampling_params)

Premium Calibration

This model was quantized using TevunahAi's premium multi-dataset calibration process:

Calibration Details

• Total Samples: 2,048 (4-8x industry standard)
• Datasets Used: 4 complementary sources
• Coverage: Comprehensive across all use cases

Dataset	Samples	Purpose
Open-Platypus	512	STEM reasoning and logic
UltraChat-200k	512	Natural conversations
OpenHermes-2.5	512	Instruction following
SlimOrca	512	Diverse general tasks

Why Premium Calibration?

Most FP8 quantizations use 128-512 samples from a single dataset. TevunahAi uses 2,048 samples across 4 diverse datasets, ensuring:

• ✅ Superior robustness across task types
• ✅ Better statistical coverage for quantization scales
• ✅ Minimal quality loss compared to FP16
• ✅ Production-grade reliability

When quality matters, choose TevunahAi FP8 quantizations.

Quantization Details

• Target Layers: All Linear layers except lm_head
• Precision: FP8 (E4M3 format)
• Hardware Requirements: NVIDIA Ada Lovelace or Hopper (native FP8) or Ampere with emulation

Quantization Infrastructure

Quantized on professional hardware optimized for high-quality model compression:

• CPUs: Dual Intel Xeon Max 9480 (224 threads, 128GB HBM2e @ 2000 GB/s)
• Memory: 256GB DDR5-4800 (16 DIMMs, 8-channel per socket, ~614 GB/s)
• Total Memory Bandwidth: ~2,614 GB/s aggregate
• GPU: NVIDIA RTX 5000 Ada Generation (32GB VRAM) with native FP8 support
• Software: Ubuntu 25.10 | Python 3.12 | PyTorch 2.8 | CUDA 13 | llm-compressor

This infrastructure enables rigorous multi-dataset calibration and quality validation for production-grade quantizations.

Performance Notes

• Quantization time: 58.2 minutes
• Calibration: 2,048 samples (premium quality)
• Memory reduction: 16GB FP16 → ~8GB FP8 (50% reduction)
• Inference speed: 2-3x faster on Ada Lovelace GPUs vs FP16

About Apertus

Apertus-8B is a high-quality 8B parameter instruction-tuned model by Swiss AI, known for:

• Strong reasoning capabilities
• Multilingual support
• Efficient architecture
• Apache 2.0 license

License

Apache 2.0 (same as original model)

Credits

• Original model by Swiss AI
• Quantized by TevunahAi
• Quantization powered by llm-compressor

---

Why TevunahAi FP8?

Uncompromising Quality

Aspect	Standard FP8	TevunahAi Premium FP8
Calibration Samples	128-512	2,048
Datasets	Single	4 diverse
Quality Validation	Basic	Rigorous
Use Case	General	Production-grade

Professional Infrastructure

• 2.6 TB/s memory bandwidth (dual Xeon Max + octa-channel DDR5)
• Multi-dataset calibration protocol
• Quality-first approach
• Enterprise-ready results

TevunahAi: The gold standard for FP8 quantizations.