Instructions to use infly/Infinity-Parser2-Pro with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use infly/Infinity-Parser2-Pro with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="infly/Infinity-Parser2-Pro")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("infly/Infinity-Parser2-Pro")
model = AutoModelForImageTextToText.from_pretrained("infly/Infinity-Parser2-Pro")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use infly/Infinity-Parser2-Pro with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "infly/Infinity-Parser2-Pro"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "infly/Infinity-Parser2-Pro",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/infly/Infinity-Parser2-Pro

SGLang

How to use infly/Infinity-Parser2-Pro with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "infly/Infinity-Parser2-Pro" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "infly/Infinity-Parser2-Pro",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "infly/Infinity-Parser2-Pro" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "infly/Infinity-Parser2-Pro",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use infly/Infinity-Parser2-Pro with Docker Model Runner:
```
docker model run hf.co/infly/Infinity-Parser2-Pro
```

Troubleshooting flash-attn==2.8.3 Installation Issues

by KexuanRen - opened Apr 16

Discussion

KexuanRen

inftech.ai org Apr 16

•

edited Apr 16

Deep Dive into flash-attn==2.8.3 Installation Pitfalls and Solutions during Deployment

When deploying Infinity-Parser2-Pro, installing flash-attn is often the most significant bottleneck in environment setup. While the standard recommendation is pip install flash-attn==2.8.3 --no-build-isolation, developers frequently encounter two major roadblocks:

Missing Official Pre-built Wheels (Environment Mismatch)
Problem: For cutting-edge stacks like CUDA 12.x + PyTorch 2.10 + Python 3.12, the official PyPI repository may lack pre-compiled wheels. This forces pip to attempt building from source, which is time-consuming and prone to failure.
Solution: To bypass complex build dependencies, it is highly recommended to use community-maintained third-party wheels tailored for these specific environments.

Recommended Steps:

Download the file using wget (or download locally and upload to your server):
wget https://github.com/lesj0610/flash-attention/releases/download/v2.8.3-cu12-torch2.10-cp312/flash_attn-2.8.3%2Bcu12torch2.10cxx11abiTRUE-cp312-cp312-linux_x86_64.whl
Install locally: Replace the path with the directory where you saved the file:
pip install /path/to/flash_attn-2.8.3+cu12torch2.10cxx11abiTRUE-cp312-cp312-linux_x86_64.whl

Out-of-Memory (OOM) Errors during Source Compilation
Problem: Compiling flash-attn from source requires significant RAM due to heavy C++ template meta-programming. If the system memory is insufficient (often < 32GB) or if too many CPU cores are compiling in parallel, the nvcc process will likely be terminated by the OS (Killed).
Solution:

Hardware: Ensure the server has sufficient physical RAM or increase the Swap space before installation.

Environment: Limit the number of concurrent build jobs by setting the MAX_JOBS environment variable (e.g., export MAX_JOBS=4) to cap peak memory usage.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment