Instructions to use athirdpath/Iambe-Storyteller-20b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use athirdpath/Iambe-Storyteller-20b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="athirdpath/Iambe-Storyteller-20b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("athirdpath/Iambe-Storyteller-20b")
model = AutoModelForCausalLM.from_pretrained("athirdpath/Iambe-Storyteller-20b")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use athirdpath/Iambe-Storyteller-20b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "athirdpath/Iambe-Storyteller-20b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "athirdpath/Iambe-Storyteller-20b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/athirdpath/Iambe-Storyteller-20b

SGLang

How to use athirdpath/Iambe-Storyteller-20b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "athirdpath/Iambe-Storyteller-20b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "athirdpath/Iambe-Storyteller-20b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "athirdpath/Iambe-Storyteller-20b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "athirdpath/Iambe-Storyteller-20b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use athirdpath/Iambe-Storyteller-20b with Docker Model Runner:
```
docker model run hf.co/athirdpath/Iambe-Storyteller-20b
```

I tested this model

by anmol989 - opened Dec 9, 2023

Discussion

anmol989

Dec 9, 2023

•

edited Dec 9, 2023

The model is not reliable. It generate good story but 1 out of 10 time. This model is ranked #9 at the ayumi benchmark means it understand character card very well and has potential but only if it was reliable. And why this gem is missing from this model - https://huggingface.co/Undi95/Storytelling-v2.1-13B-lora (Is this because this a 20B model and 13B lora cant be added ?)

athirdpath

Owner Dec 9, 2023

Yeah, 13b LORAs can't be added to 20b models, unfortunately. I'd have to go back to the first stages of Iambe where it had not yet been trained as a unified model, and inject it there. Thankfully I did (it's part of one of parent models), and the LoRA carried on into the 20b model's vastness!

Yeah, it's unstable because of the bizarre mad science I did here with task_arithmetic, the next version that is directly trained ought to fix it.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment