Instructions to use Qwen/Qwen2-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen2-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/Qwen2-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-7B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-7B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/Qwen2-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen2-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen2-7B

SGLang

How to use Qwen/Qwen2-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen2-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen2-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen2-7B with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen2-7B
```

TypeError: 'NoneType' object cannot be interpreted as an integer

by tanliboy - opened Jun 15, 2024

Discussion

tanliboy

Jun 15, 2024

Hi Qwen2 team,

I am trying to run Zephyr DPO recipe (https://github.com/huggingface/alignment-handbook/tree/main/recipes/zephyr-7b-beta) to fine-tune this model but consistently running into this error. (The SFT training works fine). Does this model use a special checkpoint configuration I need to configure? Any thoughts on the potential reason?

" [rank6]: TypeError: 'NoneType' object cannot be interpreted as an integer
[rank5]: Traceback (most recent call last):
[rank5]: File "/home/litan/alignment-handbook/scripts/run_dpo.py", line 261, in
[rank5]: main()
[rank5]: File "/home/litan/alignment-handbook/scripts/run_dpo.py", line 214, in main
[rank5]: train_result = trainer.train(resume_from_checkpoint=checkpoint)
[rank5]: File "/opt/conda/envs/handbook/lib/python3.10/site-packages/transformers/trainer.py", line 1850, in train
[rank5]: return inner_training_loop(
[rank5]: File "/opt/conda/envs/handbook/lib/python3.10/site-packages/transformers/trainer.py", line 2165, in _inner_training_loop
[rank5]: for step, inputs in enumerate(epoch_iterator):
[rank5]: File "/opt/conda/envs/handbook/lib/python3.10/site-packages/accelerate/data_loader.py", line 454, in iter
[rank5]: current_batch = next(dataloader_iter)
[rank5]: File "/opt/conda/envs/handbook/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in next
[rank5]: data = self._next_data()
[rank5]: File "/opt/conda/envs/handbook/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
[rank5]: data = self._dataset_fetcher.fetch(index) # may raise StopIteration
[rank5]: File "/opt/conda/envs/handbook/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
[rank5]: return self.collate_fn(data)
[rank5]: File "/opt/conda/envs/handbook/lib/python3.10/site-packages/trl/trainer/utils.py", line 338, in call
[rank5]: to_pad = [torch.LongTensor(ex[k]) for ex in features]
[rank5]: File "/opt/conda/envs/handbook/lib/python3.10/site-packages/trl/trainer/utils.py", line 338, in
[rank5]: to_pad = [torch.LongTensor(ex[k]) for ex in features]
[rank5]: TypeError: 'NoneType' object cannot be interpreted as an integer
[2024-06-15 02:51:57,401] [INFO] [utils.py:802:see_memory_usage] After initializing ZeRO optimizer"

tanliboy

Jun 18, 2024

•

edited Jun 21, 2024

In case anyone runs into the same problem, I figured out it is related the inconsistence between bos_token_id and bos_token.
I worked around it by changing
"bos_token": null to be "bos_token": <|endoftext|> in the tokenizer_config.json file.

jklj077

Qwen org Aug 14, 2024

please also refer to this comment. it is not needed to change the config file after the related PR in trl.

jklj077 changed discussion status to closed Aug 14, 2024

willieseun

Apr 9, 2025

Thanks a lot for figuring it out. I was looking for a solution for hours.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment