Instructions to use sevren-ai/Embla1-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use sevren-ai/Embla1-bf16 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="sevren-ai/Embla1-bf16", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("sevren-ai/Embla1-bf16", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use sevren-ai/Embla1-bf16 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "sevren-ai/Embla1-bf16" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sevren-ai/Embla1-bf16", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/sevren-ai/Embla1-bf16
- SGLang
How to use sevren-ai/Embla1-bf16 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "sevren-ai/Embla1-bf16" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sevren-ai/Embla1-bf16", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "sevren-ai/Embla1-bf16" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sevren-ai/Embla1-bf16", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use sevren-ai/Embla1-bf16 with Docker Model Runner:
docker model run hf.co/sevren-ai/Embla1-bf16
sevren-ai/Embla1-bf16
Embla 1 (sevren.ai) is a mixture-of-experts language model designed for phones. This repository contains the bf16 checkpoint.
The 4-bit format is the intended precision. Embla 1 was trained quantization-aware for 4-bit weights (affine, group size 64), and the model that runs on-device is the 4-bit release: sevren-ai/Embla1-4bit (MLX). The bf16 weights provided here are the master checkpoint from which those 4-bit weights are rounded.
Model details
- Parameters: 10B total, including 3B engram parameters; 1B active per token.
- Context length: 256K tokens (262,144).
- License: MIT.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
path = "sevren-ai/Embla1-bf16"
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True, dtype=torch.bfloat16).to(device)
inputs = tok.apply_chat_template([{"role": "user", "content": "Hi!"}], add_generation_prompt=True,
return_tensors="pt", return_dict=True).to(device)
out = model.generate(**inputs, max_new_tokens=256)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:]))
- Only
transformers(5.9 or newer) andsafetensorsare required; the modeling code ships in this repository. The model can be loaded in bf16 or fp32. - The modeling code is an eager PyTorch reference implementation. It is correct anywhere PyTorch runs, but it is not optimized for speed.
generate()maintains a KV cache, and the prompt pass runs only its last token through the second half of the layers. - Padding is not supported: use a single prompt per call, or batch rows of equal length.
- The chat format is ChatML (via
apply_chat_template). The model first reasons in a<think>block, then produces its answer.
- Downloads last month
- 119