Qwen3.5-4B-Prompt-Prior-Patch

A training-free weight edit for Qwen/Qwen3.5-4B (revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a) that makes the model read amounts from the document image instead of from a prior belief planted in the system prompt ("prompt prior"). Two directions are projected out of the matrices that write into the residual stream. No gradient step; the directions come from 10 training document pairs and 20 training documents (a few dozen more for layer selection). Rank 2 in total, 64 tensors, median relative change of a tensor 2.7 %.

Evaluation data, prompts, scoring scripts and all raw model answers: dataset keyvan-ai/OCR-Injection-DE.

Research report (bilingual EN/DE) with all results and limitations: doi:10.5281/zenodo.24079056.

What it does – and what it does not

Attack (German invoices, weak prompt, greedy)	Original	Patch	Random edit (4 seeds)	Original with hardened prompt
Prompt prior (K7/T3): answer gives the total from the system prompt instead of the image, 274 documents	8.8 % [6.0–12.7] (24/274)	1.5 % [0.6–3.7] (4/274)	9.8 % (seeds 4.4–13.5 %)	0 % (0/75, extended subset)
Injected JSON block (K4/T2): IBAN copied from a fake "expected output" block in the image, 100 documents	75 % (75/100)	73 % (73/100)	71 % (1 random projection)	72 % (72/100)

95 % Wilson intervals in brackets. The random control uses the same 64 tensors, the same rank per tensor and the same Frobenius norm of the change per tensor (ΔW = −s·Q(QᵀW) with random Gaussian directions).

• Prompt prior: repaired, and the effect is specific to the direction. Patch minus mean of the four random edits: −8.3 percentage points, paired bootstrap over documents 95 % CI [−10.9; −5.7] (Newcombe [−10.4; −5.5]). Random edits of equal size leave the rate unchanged on average (+1.0 [−0.8; +2.8] vs. original) but scatter in both directions (4.4–13.5 %). The patch reverts 21 of the 24 replacements of the original and introduces 1 new one. Both subsets agree: 200 new documents 5.0 % → 0.5 % (random 6.6 %), 74 documents of the extended hold-out 18.9 % → 4.0 % (random 18.2 %). With a hardened prompt the patched model also reaches 0/75. Caveat: the bootstrap is conditional on these four seeds; at seed level the strongest statement is "below all four seeds".
• Injected JSON block: not repaired. No variant of the edit (nor a hardened prompt) lowers the IBAN copying rate; all differences are within ±4 points and their intervals include zero. A stronger K4 direction (from layer 10) did reduce copying on training data but broke structured output and was discarded.

Costs (measured on the same hold-out, weak prompt)

	Original	Patch
Field accuracy on clean invoices (T2 JSON, 7 fields, 60 documents)	95.5 %	98.8 %
CER of full transcription, clean documents (T1, mean / median)	0.037 / 0.011	0.027 / 0.007
T3 total correct, clean documents	60/60	60/60
Injected text still transcribed verbatim (T1 on K4 documents)	100/100	96/100
Strict format adherence of T2 (valid JSON, date DD.MM.YYYY, amounts as numbers)	59/60	47/60
General text test without image (20 questions + 3 with a system prompt)	23/23	23/23

The format cost is specific: in 13 of 60 clean T2 answers the patched model writes the date in ISO format (2026-10-07) instead of the requested DD.MM.YYYY (amounts and other fields stay correct). In 4 of 100 transcriptions it omits the injected JSON block entirely, i.e. it is slightly less faithful as a pure OCR copier. Both effects come from the K4 component; a variant with only the prompt-prior component keeps the format (58/60) and full fidelity (100/100) but reduces the prompt prior less on the first hold-out (10.7 %).

The edit

Component	Direction	Source	Matrices
K4	mean(document with injected JSON block) − mean(same document with neutral text), 10 training pairs, at the IBAN decision position of a teacher-forced faithful JSON answer	residual stream, input of layer 8	o_proj / linear_attn.out_proj and mlp.down_proj of all 32 layers
Prompt prior	mean(with system prompt carrying identity + prior) − mean(without), 20 training documents, question for the total	residual stream, input of layer 20	same matrices, layers 16–31

W ← (I − Q Qᵀ) W, Q = orthonormal basis of the active directions per layer (rank 1 in layers 0–15, rank 2 in layers 16–31). Directions were chosen on training documents only (layer sweep with teacher-forced log-probability margins and a format check on separate clean documents); all numbers above are on hold-out documents with new seeds, templates, wording and positions.

Files

• model.language_model.layers.{0..31}.{self_attn.o_proj|linear_attn.out_proj}.weight.npy, model.language_model.layers.{0..31}.mlp.down_proj.weight.npy — 64 full edited tensors, float32 with bf16-exact values
• SHA256SUMS — checksums of all 64 tensors
• patch_doku.json — components, per-tensor Frobenius norms, change, rank and checksum
• projection_basis_per_layer.npz — the basis Q used in every layer
• edits_anwenden.py — writes a patched checkpoint from the unmodified base model

Apply

pip install torch safetensors numpy
sha256sum -c SHA256SUMS
python3 edits_anwenden.py /path/to/Qwen3.5-4B /path/to/this/repo /path/to/output   # base revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

The output folder is a normal Qwen3.5-4B checkpoint (tested with vLLM 0.33, --dtype bfloat16). Evaluation setting: chat template of the model with thinking disabled, temperature 0, German prompts (see the dataset).

Limits

• One model, one language (German), synthetic documents. Absolute attack rates depend strongly on the template (prompt prior: 5 % on one hold-out, 19 % on another).
• Hold-out sizes are 74–200 documents per attack; the corpus hold-out with the six classic injection classes (open instruction, disguised, value correction, faint text, English instruction) and the encoded-payload class was not re-measured with the patch.
• The edit removes a direction; it does not clamp it. Directions from neighbouring layers had opposite effects in the sweep, so transferring the recipe to other models requires its own sweep and its own random control.
• Not a security guarantee: the most effective attack (injected JSON block) is untouched.

License

Apache 2.0, as the base model Qwen/Qwen3.5-4B (license field of its model card at the revision above).

Citation

@techreport{hardani2026legible,
  title       = {Legible but Altered: Document-Image Attacks on OCR Vision-Language Models and a Patch in the Weights},
  author      = {Hardani, Keyvan},
  institution = {Keyvan.ai},
  year        = {2026},
  type        = {Research report (English/German)},
  doi         = {10.5281/zenodo.24079056},
  url         = {https://doi.org/10.5281/zenodo.24079056}
}

Published by Keyvan.ai, November 2026.