[DRAFT] fix: transformers 5.x compat (cache_position + kwargs naming)
Summary
This PR fixes two issues that prevent dots.ocr from working with transformers>=5.0:
1. cache_position TypeError on generation
In transformers 5.x, cache_position is no longer maintained in the generation loop. The current code does cache_position[0] == 0 which crashes with TypeError: 'NoneType' object is not subscriptable.
Fix: Use a combined check compatible with both transformers 4.x and 5.x — fall back to past_key_values is None when cache_position is unavailable.
2. _validate_model_kwargs ValueError for processor outputs
forward() uses **loss_kwargs instead of **kwargs. Transformers 5.x validation only recognizes **kwargs/**model_kwargs as catch-all params, causing processor outputs like mm_token_type_ids to fail validation.
Fix: Rename **loss_kwargs to **kwargs (functionally identical).
Backward compatibility
Both fixes maintain full backward compatibility with transformers 4.x.
Issues found with this PR (refs/pr/50)
We've been testing this PR for integration with docling and found two issues:
1. DotsVLProcessor.__init__ missing video_processor argument
In configuration_dots.py, DotsVLProcessor.__init__ calls:
super().__init__(image_processor, tokenizer, chat_template=chat_template)
But Qwen2_5_VLProcessor.__init__ expects a video_processor argument (it's declared in attributes = ["image_processor", "tokenizer", "video_processor"]). Since transformers 4.57+, ProcessorMixin.__init__ validates all attributes, causing:
TypeError: Received a NoneType for argument video_processor, but a BaseVideoProcessor was expected.
Fix: Either pass video_processor=None with a way to skip validation, or add a dummy/minimal video processor. Alternatively, override attributes in DotsVLProcessor to exclude video_processor:
class DotsVLProcessor(Qwen2_5_VLProcessor):
attributes = ["image_processor", "tokenizer"]
...
2. Predictions differ under transformers 5.x (5.5.3 / 5.8.0.dev0)
Under transformers 4.57.6, the model produces correct multi-element layout JSON with proper bounding box coordinates matching the input image dimensions.
Under transformers 5.x (both 5.5.3 and 5.8.0.dev0), the model produces garbage output: a single fullpage bbox [{"bbox": [0, 0, 1036, 1036], "category": "Picture"}] regardless of image content or size.
The prepare_inputs_for_generation compatibility code (is_prefill detection) appears correct — we traced the full generation loop and all kwargs are forwarded properly. The root cause is still unclear, but something in transformers 5.x generate() breaks the positional encoding or vision embedding pipeline.
Current workaround: We run dots.ocr/dots.mocr evaluation with transformers 4.57.6.
Suggesting to mark this PR as draft until both issues are resolved. Happy to help debug further.