Instructions to use litert-community/Qwen2-VL-2B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use litert-community/Qwen2-VL-2B with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli # A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run # (or pass its name right after the repo id). litert-lm run \ --from-huggingface-repo=litert-community/Qwen2-VL-2B \ --prompt="Write me a poem"
- LiteRT
How to use litert-community/Qwen2-VL-2B with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
- Qwen2-VL-2B-Instruct β LiteRT-LM (on-device Vision-Language Model)
- How to use
- Performance
- Galaxy S26 β GPU and CPU
- Pixel 11 Pro β GPU
- Run on iPhone / macOS
- Run on Android β Google AI Edge Gallery
- Conversion notes
- 2026-08-28 β start_token fix (weights unchanged)
- 2026-08-29 β default system prompt restored (weights unchanged)
- 2026-08-30 β tokenizer section replaced (weights unchanged)
- 2026-08-31 β chat template fix: history turns with role 'assistant' (metadata-only, weights unchanged)
- 2026-11-14 β GPU graph (weights unchanged)
- Known runtime issues
- How to use
LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).
Qwen2-VL-2B-Instruct β LiteRT-LM (on-device Vision-Language Model)
This repository is a conversion of Qwen/Qwen2-VL-2B-Instruct to LiteRT-LM, made with litert-torch (the file published on 2026-11-14, 11:21 UTC, has its prefill/decode graph rewritten for the GPU by a separate script; weights unchanged) for Google's LiteRT-LM runtime, for on-device image+text inference: on a Galaxy S26 GPU it decodes 46.3 tokens/s and prefills 1,024 tokens at 657 tokens/s (LiteRT-LM v0.19.0, 2026-11-14).
Qwen2-VL is Alibaba's widely-used general-purpose vision-language model: a dynamic-resolution ViT vision encoder feeds the Qwen2-1.5B language decoder, giving strong general VQA, description, and document/OCR reading in a phone-sized package. This file runs the whole thing through LiteRT-LM's fast_vlm multimodal path β give it an image and a question, get a grounded answer, fully on-device.
| File | Qwen2-VL-2B.litertlm (~1.78 GB) |
| Vision | Qwen2-VL ViT (32L, 1280-dim, full attention) made static 672Γ672 β 2304 patches β 2Γ2 merge β 576 image tokens, int8 weights |
| Adapter | PatchMerger (LN β 2Γ2 group β MLP), int8, output already at the 1536 text hidden size |
| Decoder | Qwen2-1.5B (28L, hidden 1536, GQA kv2), int4 weights (symmetric, blockwise-32 + OCTAV); int8 externalized embedder |
| Context (KV cache) | 4096 |
| Image input | resized to 672Γ672 (OpenAI-CLIP normalization baked into the encoder) |
| Base model | Qwen/Qwen2-VL-2B-Instruct (Apache-2.0) |
How to use
1. Install the runtime
pip install litert-lm
2. Run it in one command β this downloads the file, encodes your image and answers:
litert-lm run --from-huggingface-repo litert-community/Qwen2-VL-2B Qwen2-VL-2B.litertlm \
--attachment photo.jpg \
--prompt "Describe this picture in one sentence." --top-k 1
On a 640 Γ 480 photo of two tabby cats on a pink blanket, this file answers:
Two cats are sleeping on a pink blanket, with one remote control lying next to them.
(Run with the flags above on litert-lm 0.19.2 on a Mac, 2026-11-14. litert-lm 0.18.0 to 0.19.2 give the same sentence on a Mac CPU with 1 thread, and litert-lm 0.17.0 to 0.19.2 give it on a Mac GPU with the vision encoder on the GPU too, --backend gpu --vision-backend gpu. On the CPU, litert-lm 0.17.0 and 0.17.1 answer "Two cats are sleeping on a pink blanket, with one of them lying on its side and the other lying on its back.")
If run stops right after the download with no error message from the CLI (a bus error; exit status 138 on a Mac) or with an error log that contains too large for file size, the downloaded file may be incomplete. The litert-lm CLI (0.14.0 to 0.19.2, and litert-lm-nightly 0.20.0.dev20261112) can keep an incomplete download in ~/.litert-lm/cache/huggingface/litert-community/Qwen2-VL-2B/ and uses it again on later runs without checking its size. Delete that file and run the command again.
If you downloaded Qwen2-VL-2B.litertlm with --from-huggingface-repo before it was replaced on 2026-11-14 (11:21 UTC), delete ~/.litert-lm/cache/huggingface/litert-community/Qwen2-VL-2B/Qwen2-VL-2B.litertlm, then run the command again. Without deleting it, the litert-lm CLI (0.14.0 to 0.19.2) keeps running the previous file: it uses a file it finds there without checking it against this repo.
Drop --prompt for an interactive chat, and pass --attachment more than once for several images. litert-lm serve exposes the same model as a local OpenAI-compatible API. The same file runs on macOS, Linux and Windows.
Performance
Measured on an earlier file: litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine. These figures cover the text path; the vision encoder runs once per image and is not included.
| Device | Backend | Prefill (256) | Decode | TTFT |
|---|---|---|---|---|
| Apple M4 Max (macOS) | CPU | 141 tok/s | 42.7 tok/s | 1.96 s |
| Apple M4 Max (macOS) | GPU (Metal) | 1753 tok/s | 139.1 tok/s | 0.16 s |
Every desktop backend listed above was checked by actually generating a caption on it, not just by reading the benchmark tool's output.
Accuracy note
Device-verified on a Pixel 8a (Google AI Edge Gallery, vision on GPU + decoder on CPU) and on the desktop LiteRT-LM runtime (macOS CPU):
- General description / VQA (photo, on-device): accurate and detailed grounding β an Ansel-Adams-style landscape β "a striking black-and-white photograph β¦ an expansive river winds through a lush, forested valley β¦ to the left β¦ a steep, rocky hillside with rocky cliffs β¦ low hills and mountains, covered in snow, emphasizing the stark nature of the wilderness".
- Document OCR (
Read all the text in this image., macOS): perfect transcription of a synthetic report page β every figure, the e-mail address and the phone number. - Counting / reading tables works (e.g. "there are 5 products listed"); ranking table cells across rows is the one weak spot β see the M-RoPE note below.
- Vision tower: static-rewrite vs the reference implementation corr 1.0 (fp32), 0.90 at int8, zero FLEX/CUSTOM ops (GPU-clean); the int8 vision is functionally accurate on both VQA and OCR above.
- Decoder: extracted as a standalone Qwen2 model β bit-exact fp32 logits vs the original; int4 is the same blockwise-32 + OCTAV recipe used across the shipped Qwen3/Qwen2 LLMs.
M-RoPE note & one known limitation. Qwen2-VL's decoder uses 3-D M-RoPE. The LiteRT-LM
fast_vlmruntime supplies plain sequential (1-D) positions β for text tokens this is mathematically identical, and it preserves description, general VQA, counting, and OCR with no visible loss (A/B-verified against true M-RoPE). The one casualty is cross-cell comparison reasoning over 2-D structures β e.g. "which row of this table has the highest value": the 1-D positions flatten the table's 2-D layout, so the model can read every cell correctly (OCR is perfect) but may pick the wrong cell when asked to compare across rows/columns. This is inherent to the runtime (it has no M-RoPE), not the base model or the quantization β the fp32 base with true M-RoPE answers such questions correctly. Use it for reading and describing tables; don't rely on it to rank cells.
General-purpose VLM. Unlike an OCR-specialist, Qwen2-VL answers open questions about an image (describe, count, read, reason) as well as extracting text. Ask it anything about the picture β with the 2-D-table-ranking caveat above.
One image per chat. Like the other fast_vlm files, send each image in a fresh conversation β a second image in the same chat degrades (context bleed from the first turn was observed on CPU).
Galaxy S26 β GPU and CPU
Each run prefills 1,024 tokens and generates 256 tokens, three times (three iterations): the runtime's benchmark mode takes the prompt file's one sentence ("Explain what on-device AI means in simple terms.") in the chat template and pads it with token id 0 to 1,024 tokens. The GPU row is the median of iterations 2β3 over two runs. On the CPU, iteration 1 runs faster than iterations 2 and 3, so each iteration has its own row: the median of that iteration over two runs.
| Backend | Prefill, 1,024 tokens | Decode | Time to first token | Peak (VmHWM) |
|---|---|---|---|---|
GPU (LiteRT GPU, every node of the prefill and decode graphs), iterations 2β3 |
657 tok/s | 46.3 tok/s | 1.58 s | 0.77 GB |
| CPU (XNNPACK), iteration 1 | 201 tok/s | 28.0 tok/s | 5.14 s | 1.97 GB |
| CPU (XNNPACK), iteration 2 | 115 tok/s | 26.1 tok/s | 8.94 s | 1.97 GB |
| CPU (XNNPACK), iteration 3 | 115 tok/s | 26.0 tok/s | 8.93 s | 1.97 GB |
Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16), litert_lm_advanced_main built from LiteRT-LM v0.19.0 (commit 771b06e1), --benchmark with a 1,024-token prefill and 256 decode tokens, --max_num_tokens=2048, 2026-11-14. Each run is a new process. The rows cover the text path.
The phone lowers its CPU clock caps from 2β6 s into a CPU run, to 1.79 / 1.75 GHz by 16β18 s in, so iterations 2 and 3 run at a lower clock than iteration 1. On the CPU, this file's prefill and decode medians in each iteration were between 1.4 % below and 3.2 % above those of the previous file (replaced on 2026-11-14, see Conversion notes). Two counted runs of the same file differed by up to 6.6 %.
Peak is the process's VmHWM. On the CPU it was 1.97 GB in this file's two counted runs (2.01 GB in a warm-up run before them) and 2.07β2.09 GB in the previous file's two. VmHWM does not count memory the GPU driver holds: the phone's GPU allocation counter (kgsl_page_alloc) rose by 1.95β1.96 GB during this file's GPU runs and by 1.78β1.79 GB during those of the previous file.
With a photo of two cats on a pink blanket and the vision encoder on the GPU (--vision_backend=gpu), the phone answered "Describe this picture in one sentence." with "Two cats are sleeping on a pink blanket, with one remote control lying next to them."
On this phone the GPU decodes 1.7β1.8Γ as fast as the CPU and prefills 3.3β5.7Γ as fast, so this repo's litertlm_manifest.json recommends gpu for flagship-class Android. That recommendation rests on this phone alone: on the Pixel 11 Pro below, the CPU was not measured.
The runs behind each number, and the S26 section this card carried for an earlier file: MEASUREMENTS.md. GPU wiring, including the Gallery import toggle: GPU guide.
Pixel 11 Pro β GPU
On a Pixel 11 Pro GPU (PowerVR, Android 17, the same LiteRT-LM v0.19.0 build and workload as on the Galaxy S26), this file decodes 18.6 tok/s, 15 % faster than the previous file, and prefills 532 tok/s, 2 % faster.
On the Pixel 11 Pro, one of the 8 short questions, "Which is larger: 0.9 or 0.11? Answer briefly.", got a different answer from this file when the run found the GPU cache files that an earlier run of the same file had written next to the model: "0.11 is larger." in 5 of 5 such runs, and "0.9 is larger than 0.11." in 4 of 4 runs that started without them. The previous file answered "0.9 is larger than 0.11." with and without its cache files (4 of 4 and 3 of 3), and so did this file on the Galaxy S26 (one run with the files and one without). The other 7 questions passed with both files. The runs: MEASUREMENTS.md.
Run on iPhone / macOS
Use the LiteRT-LM Swift runtime (swift-litert-lm). Load Qwen2-VL-2B.litertlm with the vision tower enabled (Modality.textImage), attach a photo, and ask a question.
Vision-only file (no audio tower): bring the engine up with the vision modality only β requesting
.allfails at session creation on files without an audio section.
Run on Android β Google AI Edge Gallery
Install a recent Google AI Edge Gallery, download Qwen2-VL-2B.litertlm, import it (tap +, enable "Support image"), attach an image and ask.
For sample apps, see google-ai-edge/litert-samples.
Conversion notes
- LiteRT-LM
fast_vlmfile: VISION_ENCODER ([1,672,672,3]β[1,2304,1280]) + VISION_ADAPTER ([1,2304,1280]β[1,576,1536]) + single-token EMBEDDER + PREFILL_DECODE (embeddings-input). - Static rewrite of the dynamic-res vision tower: Qwen2-VL's ViT is native-resolution (packed patches, 2-D rope,
cu_seqlens) and does nottorch.export. The static graph fixes 672Γ672 and uses full attention (single image = one sequence) with precomputed 2-D rope. - Conv3d β Conv2d fold. The patch-embed is a
Conv3d(temporal_patch_size=2); for a single image the processor duplicates it into the 2 temporal frames, so the Conv3d over 2 identical slices equals aConv2dwith the summed temporal kernelw[:,:,0]+w[:,:,1]β GPU-safe, no Conv3d op. - No GATHER_ND (this is what lets it run on the phone GPU). The obvious way to reorder raster patches into the merger's 2Γ2-block order is a
gatherβ but that emits aGATHER_NDop, which the mobile GPU delegate cannot compile, so the vision executor fails to create and the whole engine won't load. Instead, patches stay in raster order through the encoder (full attention is permutation-equivariant, so ordering is irrelevant as long as each patch carries its own 2-D rope), and the 2Γ2 merge is done in the adapter with 4 strided slices + concat (f[:,0::2,0::2] β¦ f[:,1::2,1::2]), all ops β€4D. Static-rewrite corr vs the reference stays 1.0. - Decoder: the Qwen2-1.5B text model inside the VLM is re-hosted as a standalone
Qwen2ForCausalLM(state-dict 1:1, bit-exact logits, lm_head tied to embeddings) and exported with the standard litert-torch path, cache 4096.
2026-08-28 β start_token fix (weights unchanged)
The file's LlmMetadata start_token held the literal string "None". This tokenizer has no BOS, and LiteRT-LM resolved that string to a real vocabulary token β so every prompt began with the word None, which the model was never trained on. The start token has been removed.
Metadata-only change: every section of the file except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file β only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28 (03:19 UTC), download again; with the litert-lm CLI, first delete the cached file as described under the command in How to use.
2026-08-29 β default system prompt restored (weights unchanged)
The upstream chat template emits a default system turn whenever the caller sends no system message β for this model: You are a helpful assistant.. The converter's template check renders the template with a system message already present, so that block was never seen and never reached the file: with no system message the model was running without the default system turn it was tuned with. The chat template in Qwen2-VL-2B.litertlm now emits the block exactly once when no system message is given. In Qwen2-VL-2B.litertlm, the block is not emitted when you pass a system message. The restored block adds 11 prefill tokens to a conversation that sends no system message, so time-to-first-token grows by that much; per-token speed is unchanged.
Metadata-only change: every section of the file except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file per token β only the file's own sha256 differs. What changed is the input: with no system message, the prompt now renders byte-identical to the upstream chat template's output, verified on the LiteRT-LM runtime. A system message you pass yourself renders as before. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before this file reached the Hub on 2026-08-28 (21:37 UTC), download again; with the litert-lm CLI, first delete the cached file as described under the command in How to use.
2026-08-30 β tokenizer section replaced (weights unchanged)
The tokenizer in Qwen2-VL-2B.litertlm was a SentencePiece conversion of the model's BPE tokenizer, and the conversion lost the byte-level semantics: a standalone accented letter or symbol (Γ©, Γ±, ΓΌ, Β°, Β·, β¦) was encoded to the id of a single-byte token instead of the token the upstream tokenizer uses, and 36 of the 223 standalone characters U+00A1βU+017F in the checks (Β‘, Δ to Δ‘, and Ε²) became the token the conversion had reused as UNK β the end-of-text token for this vocabulary. Qwen2-VL-2B.litertlm now embeds the upstream tokenizer.json (the same HF tokenizer path most files in this collection use).
Tokenizer-only change: every section of the file except the tokenizer is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the chat template are unchanged and the speed and memory numbers on this card still describe this file per token β only the file's own sha256 differs. Verified on the LiteRT-LM runtime: one user message ("What is the capital of France?") as the chat template renders it, 7 test strings, the 223 standalone characters U+00A1βU+017F and every special token now tokenize identically to the upstream tokenizer, and the four ASCII-only test questions answer byte-identically to the previous file (same ids in, same tokens out). In the checks, text with accented letters, symbols or non-Latin scripts and a short Python snippet reach the model differently from before (digits, plain English text and the test emoji do not), so individual answers to such prompts can change. When this entry was written, the accuracy figures on this card and its on-device rows had been measured on earlier files unless a row said otherwise, and the 2026-08-25 Galaxy S26 GPU check had not been re-run on the 2026-08-30 file (the runtime's tokenizer code is the same on macOS and on device; the weights and graph are byte-identical). If you downloaded before 2026-08-30, download again; with the litert-lm CLI, first delete the cached file as described under the command in How to use.
2026-08-31 β chat template fix: history turns with role 'assistant' (metadata-only, weights unchanged)
The chat template in Qwen2-VL-2B.litertlm matched history turns only on role 'model', so a conversation created through the Conversation API with the standard role 'assistant' (for example create_conversation(messages=[...]) restoring a chat) silently dropped every assistant turn from the rendered prompt β the model saw consecutive user turns with its own replies gone. The template's role conditions now also accept 'assistant'. The engine's own send_message flow was never affected.
Template-only change: every section of the file except the metadata is byte-identical to the previous file (verified by per-section sha256), and single-turn renders, token ids, prefill counts and greedy answers are unchanged (verified on the LiteRT-LM runtime), so the numbers on this card still describe this file β only the file's own sha256 differs. Re-download only matters if you inject conversation history with role 'assistant'. Details: google-ai-edge/LiteRT-LM#3417 (https://github.com/google-ai-edge/LiteRT-LM/issues/3417).
2026-11-14 β GPU graph (weights unchanged)
The prefill/decode graph now has the GPU form litert-torch writes with --apply_gpu_composites. Every weight, and every section outside that graph, is byte-identical: sha256 b6f2f7f8β¦ (1,783,463,936 B) β 5b0fa0cfβ¦ (1,783,808,000 B). On a Mac the two files give the same replies byte for byte: 4 prompts and a picture on the CPU (litert-lm 0.19.2, 1 thread, greedy decoding with top-k 1), and one prompt and a picture on the CPU and the GPU with litert-lm 0.17.0 to 0.19.2. 8 short questions on the GPU: 8 of 8 with both files on a Mac and a Galaxy S26; for the Pixel 11 Pro, see the Pixel 11 Pro section. Galaxy S26 GPU (LiteRT-LM v0.19.0, runs in the order previous, new, new, previous): decode 38.5 β 46.3 tokens/s, 1,024-token prefill 600 β 657 tokens/s. Procedure: REPRODUCE.md. Measured on this file: the opening sentence, the example answer under How to use and the paragraph after it, the Galaxy S26 and Pixel 11 Pro sections, the litert-lm 0.16.0 note and this entry. Every other measurement on this card was taken on earlier files.
Known runtime issues
External weights (since 2026-11-09). In Qwen2-VL-2B.litertlm, the weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights) instead of inside the model graph. On a Pixel 8a, the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on a Qwen3-0.6B file with external weights (LiteRT-LM#3986). The earlier file with the weights inside the graph stays at revision f9529fb.
litert-lm 0.16.0 does not start a conversation with this file or with the previous file (Failed to create conversation, from the vision encoder's executor; checked on a Mac, CPU and GPU, 2026-11-14); litert-lm 0.17.0 to 0.19.2 do.
From the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because this file holds the embedding table in its own section and the decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). One question per conversation avoids it, as does a new conversation that starts with the earlier messages and the new question.
From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders this file's chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on this file's template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.
On LiteRT-LM 0.18.1 and 0.19.1, once a message fails because it does not fit in the context window, every later message in that conversation fails as well (the next one with Task failed with state: 7). Start a new conversation after such an error.
- Downloads last month
- 5,855