Instructions to use QuantFactory/DRT-o1-7B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use QuantFactory/DRT-o1-7B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf QuantFactory/DRT-o1-7B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf QuantFactory/DRT-o1-7B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf QuantFactory/DRT-o1-7B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use QuantFactory/DRT-o1-7B-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "QuantFactory/DRT-o1-7B-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "QuantFactory/DRT-o1-7B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
- Ollama
How to use QuantFactory/DRT-o1-7B-GGUF with Ollama:
ollama run hf.co/QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use QuantFactory/DRT-o1-7B-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "QuantFactory/DRT-o1-7B-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use QuantFactory/DRT-o1-7B-GGUF with Docker Model Runner:
docker model run hf.co/QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
- Lemonade
How to use QuantFactory/DRT-o1-7B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.DRT-o1-7B-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use QuantFactory/DRT-o1-7B-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use QuantFactory/DRT-o1-7B-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf QuantFactory/DRT-o1-7B-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "QuantFactory/DRT-o1-7B-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
QuantFactory/DRT-o1-7B-GGUF
This is quantized version of Krystalan/DRT-o1-7B created using llama.cpp
Original Model Card
DRT-o1
๐ค DRT-o1-7B | ๐ค DRT-o1-8B | ๐ค DRT-o1-14B | ๐ Paper
This repository contains the resources for our paper "DRT-o1: Optimized Deep Reasoning Translation via Long Chain-of-Thought"
Updates:
- 2024.12.31: We updated our paper with more detals and analyses. Check it out!
- 2024.12.31: We released the testing set of our work, please refer to
data/test.jsonl - 2024.12.30: We released a new model checkpoint using Llama-3.1-8B-Instruct as the backbone, i.e., ๐ค DRT-o1-8B
- 2024.12.24: We released our paper. Check it out!
- 2024.12.23: We released our model checkpoints. ๐ค DRT-o1-7B and ๐ค DRT-o1-14B.
If you find this work is useful, please consider cite our paper:
@article{wang2024drt,
title={DRT-o1: Optimized Deep Reasoning Translation via Long Chain-of-Thought},
author={Wang, Jiaan and Meng, Fandong and Liang, Yunlong and Zhou, Jie},
journal={arXiv preprint arXiv:2412.17498},
year={2024}
}
Quick Links
Introduction
In this work, we introduce DRT-o1, an attempt to bring the success of long thought reasoning to neural machine translation (MT). To this end,
- ๐ We mine English sentences with similes or metaphors from existing literature books, which are suitable for translation via long thought.
- ๐ We propose a designed multi-agent framework with three agents (i.e., a translator, an advisor and an evaluator) to synthesize the MT samples with long thought. There are 22,264 synthesized samples in total.
- ๐ We train DRT-o1-8B, DRT-o1-7B and DRT-o1-14B using Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct as backbones.
Our goal is not to achieve competitive performance with OpenAIโs O1 in neural machine translation (MT). Instead, we explore technical routes to bring the success of long thought to MT. To this end, we introduce DRT-o1, a byproduct of our exploration, and we hope it could facilitate the corresponding research in this direction.
Models
Model Access
| Backbone | Model Access | |
|---|---|---|
| DRT-o1-7B | ๐ค Qwen2.5-7B-Instruct | ๐ค DRT-o1-7B |
| DRT-o1-8B | ๐ค Llama-3.1-8B-Instruct | ๐ค DRT-o1-8B |
| DRT-o1-14B | ๐ค Qwen2.5-14B-Instruct | ๐ค DRT-o1-14B |
Model Performance
| GRF | CometKiwi | GRB | BLEU | CometScore | |
|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | 79.25 | 70.14 | 73.30 | 18.55 | 74.58 |
| Qwen2.5-7B-Instruct | 81.53 | 70.36 | 77.92 | 27.02 | 76.78 |
| Qwen2.5-14B-Instruct | 84.74 | 72.01 | 80.85 | 30.23 | 78.84 |
| Marco-o1-7B | 82.41 | 71.62 | 77.50 | 29.48 | 77.41 |
| QwQ-32B-preview | 86.31 | 71.48 | 83.08 | 27.46 | 78.68 |
| DRT-o1-8B | 84.49 | 70.85 | 80.80 | 32.67 | 78.81 |
| DRT-o1-7B | 85.57 | 71.78 | 82.38 | 35.54 | 80.19 |
| DRT-o1-14B | 87.19 | 72.11 | 83.20 | 36.46 | 80.64 |
Model Prompts
During model inference, please use the following prompts:
- System prompt:
You are a philosopher skilled in deep thinking, accustomed to exploring complex problems with profound insight. - User prompt:
Please translate the following text from English to Chinese:\n[An English text]
DRT-o1 models will first generate the thought and then provide the final translation, with the following format:
<thought>
[Reasoning process]
</thought>
<output>
[Final translation]
</output>
Quickstart
- โท๏ธ Huggingface Transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "Krystalan/DRT-o1-7B"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
prompt = "Please translate the following text from English to Chinese:\nThe mother, with her feet propped up on a stool, seemed to be trying to get to the bottom of that answer, whose feminine profundity had struck her all of a heap."
messages = [
{"role": "system", "content": "You are a philosopher skilled in deep thinking, accustomed to exploring complex problems with profound insight."},
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
**model_inputs,
max_new_tokens=2048
)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
- โท๏ธ vllm:
Deploying LLMs:
python3 -m vllm.entrypoints.openai.api_server --model [model_ckpt] --served-model-name [model_name]
Calling LLMs:
from openai import OpenAI
# Set OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
chat_response = client.chat.completions.create(
model=[model_name],
messages=[
{"role": "system", "content": "You are a philosopher skilled in deep thinking, accustomed to exploring complex problems with profound insight."},
{"role": "user", "content": "Please translate the following text from English to Chinese:\nThe mother, with her feet propped up on a stool, seemed to be trying to get to the bottom of that answer, whose feminine profundity had struck her all of a heap."},
],
temperature=0.1,
top_p=0.8,
max_tokens=2048,
extra_body={
"repetition_penalty": 1.05,
},
)
print("Chat response:", chat_response)
Translation Cases
| Source Sentence | Qwen2.5-14B-Instruct | QwQ-32B-preview | DRT-o1-14B |
|---|---|---|---|
| Women were weeping and children crying, and all were going as fast as seemingly lay in their power, looking behind now and then as if pursued by some deadly enemy. | ๅฆๅฅณๅจๅญๆณฃ๏ผๅญฉๅญไปฌๅจๅๅท๏ผๆๆไบบ้ฝไผผไนๅจๅฐฝๅ จๅๅฟซ้ๅ่ก๏ผไธๆถๅๅคดๅผ ๆ๏ผไปฟไฝ่ๅๆๆไธช่ดๅฝ็ๆไบบๅจ่ฟฝ่ตถใ | ๅฆๅฅณไปฌๅญๆณฃ็๏ผๅญฉๅญไปฌๅญๅ็๏ผๆฏไธชไบบ้ฝๅจๅฐฝๅๅ ๅฟซ่ๆญฅ๏ผไธๆถๅๅคดๅผ ๆ๏ผไปฟไฝ่ๅๆ่ดๅฝ็ๆไบบๅจ่ฟฝๅปใ | ๅฆๅฅณๆฒๆณฃ๏ผๅญฉ็ซฅๅๅท๏ผไผไบบ็พๅฅ๏ผไธๆถๅ้ฆ๏ผไปฟไฝๆ่ดๅฝไนๆๅจๅ็ดง่ฟฝใ |
| Wait through this season of the rain Till suns of autumn dry the plain, Then shall thy giant foe, and all His host and realm, before thee fall. | ็ญๅพ ่ฟ้จๅญฃ่ฟๅป๏ผ็ดๅฐ็งๆฅ็้ณๅ ๆๅนฒๅ้๏ผ้ฃๆถไฝ ็ๅทจๆๅๅ ถ้จไผๅ็ๅฝ้ฝๅฐๅๅจไฝ ้ขๅใ | ๅฟ่ฟ้จๅญฃๆผซ้ฟๅค๏ผๅพ ็ง้ณๅ็ฉบ๏ผๆๅนฒๅคงๅฐๆนฟๆถฆ่ฒ๏ผ้ฃๆถ๏ผๅทจๆๅฎๅฐ๏ผไป็ๅ้ๅ็ๅฝ๏ผๅ จๅจไฝ ่ไธใ | ๅๅฐฝ่ฟ้จๅญฃ๏ผๅพ ็งๆฅ้ณๅ ้ฉฑๆฃๅนณๅไธ็ๆๅไธไธๆนฟๆถฆ๏ผ้ฃๆถ๏ผไฝ ้ฃๅทจๆ๏ผ่ฟๅไป็ๅ้ไธ็ๅฝ๏ผ้ฝๅฐๅๅๅจไฝ ่ไธใ |
| And if their words had taken another turn...if he himself had been less fastidious about intruding on another man's secrets...it was cruel to think how thin a film had shut out rescue from all this guilt and misery. | ่ๅฆๆไปไปฌ็่จ่พ่ฝฌๅไบๅฆไธไธชๆนๅโฆโฆๅฆๆไป่ชๅทฑๅฏนไพตๆฐไปไบบ็งๅฏไธ้ฃไนไปๆโฆโฆๆณๅฐๆญฃๆฏ้ฃๅฑ่่็้้ๅฐๆๆดๆไน้จๅค๏ผไฝฟๅพ่ฟไธๅ็็ฝชๆถๅ็่ฆๆ ๆณ้ฟๅ ๏ผ็ๆฏๆฎ้ ทใ | ่ไธ๏ผๅๅฆไปไปฌ็่ฐ่ฏ่ตฐๅไธๅโฆโฆๅๅฆไป่ชๅทฑไธๅฏนๅนฒๆถไปไบบ็งไบๅฆๆญค่ฐจๆ โฆโฆ้ฃๆถไปไผๆ่ฏๅฐ๏ผๅช่ฆไธๅฑ่่็ๅฑ้ๅฐฑ้ป้ไบๆๆด๏ผไฝฟๅพๆๆ่ฟไบ็ฝชๆถๅ็่ฆๆ ๆณ่งฃ่ฑ๏ผ่ฟ็ๆฏๆฎๅฟใ | ่ฅไปไปฌ็่จ่พ็จไฝ่ฝฌๅโฆโฆ่ฅไปๆฌไบบไธ้ฃไนไปๆไพตๅ ฅไปไบบ็ๅฟๆโฆโฆๆณๅฐ่ฟๅฑ่่็้้็ซๅฐๆๆๅ ็ไธ็่ฆ้็ฆปไบๆๆดไนๅค๏ผๅฎไธบๆฎๅฟใ |
| This cold officer upon a monument, who dropped epithets unconcernedly down, would be finer as a dead man, he thought. | ไป่ฎคไธบ๏ผ่ฟไธช็ซๅจ็บชๅฟต็ขไธ็ๅทๆผ ๅฎๅ๏ผ่ฅๆญปไบไผๆดๅฅฝ๏ผไปไธๅธฆไปปไฝๆๆ ๅฐๆไธไบไธไบ็งฐๅผใ | ่ฟไธชๅทๅฐๅฐ็ๅฎๅ็ซๅจ็บชๅฟต็ขไธ๏ผๆฏซไธๅจๆๅฐๆไธไธไบ็งฐๅท๏ผไปๆณ๏ผๅฆๆไฝไธบไธไธชๆญปไบบไผๆดๅบ่ฒใ | ็บชๅฟต็ขไธ็ๅทๆทกๅฎๅ๏ผๆผซไธ็ปๅฟๅฐๅๅ้ฃไบไฟฎ้ฅฐ่ฏญ๏ผไปๅฟๆณ๏ผ่ฅๅไธบไบก่ ๏ผๆ่ฎธๆดๆพๅฐ่ดตใ |
Data
We release the testing set of our work, please refer to data/test.jsonl, where en indicates the English source sentences, and zh denotes the corresponding Chinese translation.
We will release the long-thought MT data as well as the data collection codes soon!
License
This work is licensed under cc-by-nc-sa-4.0
- Downloads last month
- 366
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
docker model run hf.co/QuantFactory/DRT-o1-7B-GGUF: