M_T1 — Fully Frozen Vision-Language Alignment
A fully frozen-backbone image-text retrieval model: 47.2M deployed parameters, of which 2.9M are trainable.
Research artifact. Not licensed for commercial use. See NOTICE.
Code: https://github.com/Pranav210901/Efficient-VLM
What this is
A frozen DINOv3 ViT-S/16 vision tower (224 px) and a frozen all-MiniLM-L6-v2 text tower, joined by learned token aggregation and residual projections into a shared normalised 384-d space. MobileCLIP2-S0 provided a distillation signal during training only and is not part of this model.
Both backbones are completely frozen. The trainable set is a two-block 256-d learned vision-token aggregator (C4), a 128-d four-head learned-query text aggregator, two residual projection heads, and the logit scale.
| Deployed parameters | 47,196,549 |
| Trainable at inference | 2,896,389 (6.1%) |
| Seed | 43 |
| Embedding dim | 384 |
| Image size | 224 |
Results
Flickr30k Karpathy test, opened once after the model was frozen, with no training of any kind. This checkpoint (seed 43) scores 51.90% mean bidirectional R@1.
Three-seed means for context:
| Model | Flickr30k test R@1 | Params |
|---|---|---|
| M_T1 (fully frozen) | 52.90 ± 0.87 | 47.2M |
| FreezeShift dual | 62.20 ± 0.45 | 49.2M |
| OpenCLIP ViT-B/32 | 68.22 | 151.3M |
| MobileCLIP2-S0 | 78.25 | 74.8M |
| SigLIP2 ViT-B/32 | 80.46 | 376.9M |
This model does not beat the compact references. MobileCLIP2-S0 is smaller than OpenCLIP and scores higher than both models released here. The contribution is a measured boundary on frozen-encoder alignment under a fixed latency budget, not a state-of-the-art retrieval model.
Zero-shot classification transfer is weak (CIFAR-100 ~36-38%, Oxford-IIIT Pet ~8-10%, EuroSAT ~22-24%) and should be treated as a negative result.
Seed selection
The seed was chosen on the validation split, never on test. Selecting on test would have picked a different seed and inflated the reported figure.
Checkpoint provenance
This exports the COCO-dev-selected epoch. The headline validation figures in the dissertation (54.487% / 63.416%) use Flickr-validation epoch selection; under the deployed selection the same models score 54.36% / 63.04% on validation. The test figures above are unaffected by this distinction.
Files
| file | contents |
|---|---|
model.safetensors |
full stack, frozen towers included — self-contained |
adapter.safetensors |
trainable parameters only; frozen towers fetched at load |
config.json |
frozen architecture and data config |
training_recipe.json |
training recipe as run |
fingerprint.json |
provenance hashes (code, config, split, digest) |
load_model.py |
loader for both variants |
The adapter-only variant contains no third-party weights, which avoids redistributing DINOv3 entirely if that matters for your use.
Usage
# requires the repo code on PYTHONPATH
from load_model import load
model = load(".", adapter_only=False) # or True for the adapter variant
image_embeds = model.encode_image(pixel_values) # (B, 384), L2-normalised
text_embeds = model.encode_text(["a dog on a beach"])
Licensing
| Component | Terms |
|---|---|
| DINOv3 ViT-S/16 | Meta DINOv3 License — copy included as LICENSE-DINOv3.md |
| all-MiniLM-L6-v2 | Apache-2.0 |
| MobileCLIP2-S0 (teacher, training only) | Apple ML Research Model License — research only |
The DINOv3 License requires that derivatives be distributed under its terms with a copy of the Agreement provided, that publications acknowledge DINO Materials, and that use comply with Trade Controls. It prohibits military, weapons, espionage, and nuclear applications. MobileCLIP2-S0 weights are not redistributed here.
Citation
Pokhrel, P. Efficient Frozen Vision-Language Alignment. MSc dissertation, University of Surrey, 2026.
- Downloads last month
- 6