One Adapter, Both Modalities: Field Notes from Building and Serving a Multimodal Reranker

How we built LightOn-rerank, a 2B model that reranks both text passages and document pages, why we couldn't borrow the usual speedups from text reranking, and a few smaller surprises along the way!

We release a Qwen3.5-2B LoRA that reranks both text passages and document page images using a listwise setting. It reaches 62.66 NDCG@10 on ViDoRe V3, improving over the ColQwen2.5 first stage by 7 nDCG points, leading every open 2B-class multimodal reranker we measured, and staying competitive on text-only BEIR. The findings we didn't expect matter more than the leaderboard: at 2B, the ViT encoder, not the decode loop, was our largest per-window cost. All the speedups that work by splitting the listwise comparison into smaller subsets (tournament scheduling, scoring each document independently) failed: the model ranks by comparing candidates against each other, and those tricks remove the comparison. What did work is, unsurprisingly, shrinking the pool: reranking the top-20 candidates instead of 100 keeps 85% of the rerank lift at 5× fewer windows (top-10: 58% at 12× fewer), measured over the full benchmark. Finally, we explore scaling the recipe at 0.8B and 4B and observed that, while pointwise plateaus with size, listwise keeps scaling: at 4B it reaches 64.69 NDCG@10 on ViDoRe V3, edging past the Qwen3-VL-Reranker-8B (64.23 on our protocol) at half the parameters.

🤗 Models: LightOn-rerank Collection

---

Table of Contents

• Introduction
  • Why one model for both modalities
  • Two ways to rerank: pointwise vs listwise
  • The landscape we're shipping into
• Evaluation: the Endgame protocol
• Training recipe
  • Data
  • Loss
  • Cross-document attention is the rerank decision
• Results
  • ViDoRe V3 results
  • BEIR Results
• Digging deeper into Vision Results
  • Did we just overfit to one first-stage?
  • The Cross-Lingual Transfer
  • Scaling and recipe: 0.8B / 2B / 4B × pointwise / listwise
  • Results across the grid
  • A non-obvious training detail from scaling down
• Why text-reranker speedup tricks did not transfer
  • Tournament scheduling
  • First-token readout
  • Where multimodal inference actually spends its time
    • The same profile, across the whole grid
    • What this means in practice
    • The cheaper lever: rerank fewer candidates
• Recommendations, if you're building something similar
• Citation
• Acknowledgements
• Bibliography & further reading
• Appendix
  • Appendix: Hyperparameters
  • Appendix: ViDoRe V3 full results
  • Appendix: BEIR text reranking results

---

Introduction

Open multimodal rerankers are surprisingly rare. Text rerankers are plentiful, but for visual document reranking the list is short: our own MonoQwen2-VL-v0.1 was the first open visual reranker, and in the time since essentially only two others have been published (Qwen3-VL-Reranker and jina-reranker-m0). If you're building a retrieval pipeline that handles PDFs and web pages and structured text, you either chain two separate rerankers (one per modality) or you pick a larger model and accept the serving cost.

We think there's room for a small, unified reranker that handles both modalities well enough to be useful in production. This post is the field-notes version of building a successor for MonoQwen2-VL-v0.1: what landed, what didn't, and the parts we think are worth borrowing or arguing with.

Why one model for both modalities

The choice to train one joint model for both text and page inputs is a separate decision from the recipe and the size. Our previous vision reranker, MonoQwen2-VL-v0.1, was a LoRA on Qwen2-VL-2B. The LoRA setup is attractive precisely because one deployed base model can host adapters for several tasks (embedding, reranking) and swap between them. But that adapter covered document pages only, so handling text and vision meant two adapters for the same task: separate rerankers or a specialist merge, with routing and score reconciliation on top. Two LoRAs for one task defeats the point of the setup. So we trained one base + one adapter end-to-end on a mixed batch of text and page inputs. We received the expected benefits (one deployment, no routing, a single scoring scale), and consistent with what most multimodal training studies report, the two modalities helped each other rather than interfering. More details in the Results section.

Two ways to rerank: pointwise vs listwise

A reranker takes the query and the top candidates from a first-stage retriever (BM25, a dense embedder) and re-orders them with a stronger model. Everything in this post turns on the contrast between the two ways of doing that:

• Pointwise (PW). The model sees one (query, document) pair at a time and outputs a relevance score; here, score = logit("Yes") − logit("No") from a single forward pass. Every candidate is scored in isolation and the scores are sorted. The workload is embarrassingly parallel, and there is no interaction between candidates. This is what our previous vision reranker (MonoQwen2-VL-v0.1), jina-reranker-m0, and the off-the-shelf Qwen3-VL-Reranker models all do.
• Generative listwise (LW). The model sees the query and several documents together in one prompt and generates a ranking as text, e.g. [2] > [4] > [1] > [3]. Because all documents are in the same forward pass, they attend to each other: the model ranks by comparing candidates, not by scoring each one against an absolute notion of relevance. Since 100 candidates don't fit in one prompt, at deploy time you slide a window of 4 (stride 2) over the list and merge (Sun et al. 2023; Pradeep et al. 2023).

The landscape we're shipping into

Most of the explorations we did only make sense in the context of what other 2B-class multimodal rerankers did differently. Three reference points:

• Qwen3-VL-Reranker-2B (and 8B). Multi-stage pipeline trained on roughly millions of samples with large-scale supervision, hard-negative mining, and a final pointwise scoring head.
• jina-reranker-m0. Qwen2-VL-2B base with a pointwise MLP scoring head, trained on a mixed text+image dataset; production scoring is a single sigmoid per (query, document).
• RankNexus. Listwise, relying on cross-encoder distillation: candidates come from a retrieval pool with no relevance labels, so a GPT-4 / Claude-class teacher ranks them and the student is trained to reproduce the teacher's ranking.

---

Evaluation: the Endgame protocol

Before any numbers, the protocol. We call this "Endgame" because most of the headache during evaluation came from inconsistent setups across papers and HF cards. To make any of the numbers below mean something, we fixed one configuration and ran every model through it:

---

Training recipe

Base model: Qwen/Qwen3.5-2B, which natively handles both text and image inputs through a shared decoder and a ViT image encoder.

Against the three landscape reference points above, we took a different route on each axis. Listwise like RankNexus, but without an LLM teacher: as our training data includes mined hard-negatives, we use the scoring to infer the correct ranking of each group. Generative like a small RankZephyr, but multimodal and joint rather than text-only. And no pointwise scoring head, unlike Qwen's classifier and Jina's MLP; the reason is the central finding of this post: comparing large list of samples is stronger than scoring in isolation or with smaller list, and we dig into it below.

Data

• Text (107k groups): NQ + TriviaQA + MS MARCO, each example a 4-list [pos, neg_0, neg_1, neg_2]. We use the same negatives as in our DenseOn with the LateOn work: they are obtained by applying the NV-Retriever approach, retrieving the top-2048 nearest passages per query with GTE-ModernBERT, then keeping the top-50 eligible passages which similarity score falls below 99% of the query-positive similarity score as negatives. The data can be found here.
• Vision (106k groups): We use the ColPali train set with negatives mined by Nomic.
• Total: 213k groups, mixed 50/50 per microbatch.

For both text and vision, the hard-negative mining pipeline produces scores, which means the gold permutation (pos > neg_0 > neg_1 > neg_2) is constructible directly from metadata. To be precise, this is itself a form of distillation, contrary to the others that distillate the scores of a cross-encoder for the list, we distillate the ranks and use the miner directly.

Loss

Both recipes were trained on the same base model, jointly for text and vision, so every pointwise-vs-listwise comparison in this post starts from identical weights and data. Pointwise: each example is one (query, document, Yes/No) triple, and the model learns to put probability mass on Yes for matched pairs. Generative listwise: each example is one (query, [doc_1, doc_2, doc_3, doc_4]) group with a known difficulty ordering; all four documents go into a single prompt and the model is trained with cross-entropy on the permutation tokens only (ListMLE, Xia et al. 2008).

The central architectural choice for our model: the four documents must attend to each other inside a single forward pass.

Vision images resized to 512×512 (square LANCZOS) at training time. This is the smallest detail with the biggest downstream consequence: at 512×512 each page becomes ~256 image tokens (patch 16, merge 2), so four pages comfortably fit in the 2048-token vision budget alongside the prompt. The model only ever saw 512² training inputs, with whatever aspect-ratio distortion a square resize imposes on a document page. This shows up again in the resolution-sweep section below.

Full hyperparameters (LoRA rank, optimizer, scheduler, sequence lengths) are in the Appendix.

Cross-document attention is the rerank decision

In the listwise prompt, all four documents sit in the same context. Every document's representation is therefore computed while attending to the query and to the other three candidates. That is what we mean by cross-document attention: the model never answers "is this document relevant?" in a vacuum, it answers "is this document better than the others?". This section is the evidence that this comparison, not the listwise loss or the prompt format, is where the reranking quality comes from. We learned it twice: once the cheap way, once the expensive way.

The Expensive version. An earlier run used a listwise loss: groups of four documents, softmax cross-entropy over the gold ranking, but each document still scored independently with its own forward pass. On text this looked like progress but on ViDoRe V3 it went backwards. 51.86 overall NDCG@10 which is about 8.0 points below the pointwise baseline (59.87), and 3.7 points below the first stage it was reranking (55.60). It loses to the pointwise baseline on all 16 splits and to no-reranking-at-all on 11 of 16, with French hit hardest (−12.3). Same corpora, same mining pipeline, a strictly more informative training signal.

The next iteration kept the listwise loss but fed all four documents into one prompt, so the model produces a permutation as a single decoding sequence instead of four independent scores. That flipped the sign: 62.66, +2.8 over the pointwise baseline which is a swing of 10.8 NDCG points from the shared context alone, measured under one protocol. The four documents seeing each other inside the forward pass is what recovers the pointwise number and then exceeds it.

The Cheap version. The sliding window is the annoying part of serving a listwise model: 49 sequential windows per query, decode loop and all. Pointwise scoring would remove it entirely and make the whole thing parallel, which is exactly the speedup everyone wants at deploy time. So, having apparently not absorbed the expensive lesson, we paid for another edition: we took the trained listwise model and tried serving it pointwise, scoring each candidate independently and sorting. The naive readout (a Yes/No prompt) is wildly out-of-distribution for a model whose only training signal was emitting ranking permutations, so we used a more careful one that keeps the model on-template: a 2-document window against a fixed median anchor.

Concretely: for each candidate, we form the same listwise prompt the model was trained on, but with only two documents: the candidate at position [1] and a fixed "median-difficulty" anchor document at position [2]. We then read the candidate's score as logit("1") − logit("2") on the first emitted digit. The model sees the same template, the same digit vocabulary, the same query-format it saw during training; it just sees a list of length two. This is the strongest pointwise readout we could get out of these weights without fine-tuning. Same checkpoint, same input images, same first-stage candidates, just a different scoring path. The anchor is chosen per query, not per corpus: it is the document at the middle of that query's first-stage ranking (rank 50 of 100), plausibly on-topic but rarely a contender. We did not test sensitivity to anchor choice; every score is relative to this single reference document, so a different anchor, or an average over several, could shift these numbers. Treat this readout as one untuned operating point for pointwise serving, not its ceiling. On a subset of vidore v3 (computer_science_en, top-100):

We lose a big chunk of the rerank uplift over first-stage by reading the listwise model pointwise. The readout is still useful (it beats first-stage), but it is a strictly weaker product from the same weights. Both lessons are the same lesson: the strength of a listwise model is that it scores documents relative to each other. Remove the comparison, at training time (independent per-document scoring) or at serving time (pointwise readout), and the quality leaves with it.

These two results strongly highlight that the core component of a listwise reranker is that the model is able to see all the documents to rank together to judge them relatively to each other.

---

Results

ViDoRe V3 results

We compare our model against the open weights 2B-class multimodal rerankers on ViDoRe V3, one of the most challenging vision document retrieval benchmark available. It covers 8 domains (finance, computer science, HR, energy, industrial, pharmaceuticals, physics) with queries in 6 languages. We evaluate on the English and French query subsets, as those are our target languages.

ViDoRe V3 NDCG@10 (ColQwen2.5 first-stage, top-100):

The first row is the candidate pool every reranker works with: ColQwen2.5 alone gets 55.60 NDCG@10. Our pointwise model already improves over our previous vision reranker, MonoQwen2-VL-v0.1, by 2.1 nDCG points and outperforms both Qwen3-VL-Reranker-2B and jina-reranker-m0. The listwise recipe then pushes the results much further, adding another 2.8 points, largely outperforming every open model at this size and landing 7.1 points above the first stage.

Against our pointwise baseline our listwise model wins 13 of 16 splits [Three losses: energy EN (−2.1), finance_fr EN (−1.8), finance_fr FR (−1.5)]. Worth keeping in perspective: the open competitors here were trained on millions of samples or distilled from much larger teachers, while this model saw 213K groups in a single epoch. That gap hints the approach could be scaled much further.

BEIR Results

Vision is the headline, but the same checkpoint has to hold its own on plain text. Same Endgame rules on the text side: BM25 first-stage, retrieve top-100, rerank the full 100, NDCG@10. Our model runs the same W=4 stride=2 sliding window it uses for document pages; the two off-the-shelf rerankers score through their native text paths (the vLLM pooling head for Qwen3-VL-Reranker-2B, compute_score for jina-reranker-m0). 13 BEIR datasets; ⚠️ marks the two in our training mix (NQ, MSMARCO), which the clean mean excludes.

The ordering flips relative to ViDoRe V3. On vision we lead this 2B class; on text jina-reranker-m0 leads decisively (51.97 clean mean), winning 10 of the 11 clean datasets against us. Our listwise model (48.12) and Qwen3-VL-Reranker-2B (49.08) are roughly tied, splitting the clean datasets 4 to 7. Interestingly, on text our pointwise variant is slightly the stronger of our two recipes (49.13 clean mean), the mirror image of vision, where listwise wins at every size. Our own weakest split is trec-covid, where BM25 already surfaces uniformly relevant documents and re-ordering them through a 4-document window adds more noise than signal.

---

Digging deeper into Vision Results

Did we just overfit to one first-stage?

A reasonable thing to suspect after seeing the V3 leaderboard: maybe these numbers are an artifact of pairing a specific reranker with a specific first-stage retriever (ColQwen2.5). To check, we swapped only the first-stage to Qwen3-VL-Embedding-2B and re-ran the same set of rerankers under the same protocol. The candidate pool changes; everything else stays.

ViDoRe V3 NDCG@10, Qwen3-VL-Embedding-2B first-stage:

The 2B ordering is preserved: our model stays on top of Qwen3-VL-Reranker-2B and jina-reranker-m0 on both languages. The result was not an artefact of pairing with ColQwen2.5.

The Cross-Lingual Transfer

The FR columns have been sitting in the tables above without comment, and they deserve one, because the training data is English only: NQ, TriviaQA, MS MARCO, ColPali EN. The adapter never saw a French training example. ViDoRe V3 provides the same page corpora with queries written in six languages, so the French subset is a free A/B on cross-lingual transfer: identical document images, French queries. To be clear about the mechanics, there is no OCR anywhere in this pipeline; the model reads the page pixels directly.

ViDoRe V3 NDCG@10, our 2B listwise model, ColQwen2.5 first-stage, rerank-100

The Qwen backbone is natively multilingual; the LoRA carries the rerank signal in a representation space that already speaks French. (For three domains, FR beats EN: finance_fr, physics and energy)

---

Scaling and recipe: 0.8B / 2B / 4B × pointwise / listwise

The 2B listwise result is the headline of this post, but a single-cell number raises the question of whether it generalizes. We ran the full 3×2 grid on the Qwen3.5 family: three backbone sizes (0.8B, 2B, 4B) and the two recipes we already defined (pointwise, listwise). Same data, same mixed-modality batching, same evaluation protocol; each recipe keeps its own hyperparameters, unchanged across the three sizes, so the only things that vary are the backbone and the loss/scoring path. The honest caveat is that we did not re-sweep the learning rate per backbone size: each cell is the recipe transplanted as-is, not the tuned ceiling for that size, so read the grid as measuring the recipe effect rather than the best achievable number at each scale.

Results across the grid

ViDoRe V3 NDCG@10, ColQwen2.5 first-stage, rerank top-100, on the same 16 EN+FR splits:

Three insights from this grid:

• Listwise scales with size; pointwise plateaus. Pointwise 0.8B → 2B is a real jump (+11.7), but 2B → 4B is flat (−0.1). Listwise keeps climbing at every step (+4.4, then +2.0). Doubling pointwise parameters buys nothing because the model only ever decides "relevant or not" for one document at a time, and that decision saturates fast; generative listwise spends the extra capacity on richer cross-document comparisons inside the forward pass.
• The Δ column is not a clean function of size, and the reason is instructive. At 0.8B, pointwise does not just underperform: it lands below the first-stage it is reranking (48.20 vs 55.60 with no rerank). An under-capacity model scoring each document in isolation makes the ranking worse, consistent with ProRank's observation (Li et al. 2025) that small backbones struggle to reliably follow the scoring instruction at all. Cross-document attention is what keeps the 0.8B listwise model usable, hence the big +10.0 gap. From 2B up, both recipes produce working rerankers, and the gap simply tracks the scaling story: pointwise flatlines while listwise keeps climbing, so it widens again (+2.8 → +4.9).
• 4B listwise is the highest V3 number we hit: 64.69 @10. The official Qwen3-VL-Reranker-8B scores 64.23 under the same protocol, so a 4B trained on our 213K groups closes that 2× parameter gap and then some. We keep it out of the 2B leaderboard above because it changes the comparison, but it is the right context for "is size still buying us something at this point."

A non-obvious training detail from scaling down

At 0.8B, 2 epochs beat 1 epoch. The 2B rule of thumb (one epoch wins, more starts overfitting) flips. The 0.8B is under-capacity for this dataset, so the second pass still extracts signal instead of memorizing.

---

Why text-reranker speedup tricks did not transfer

Once the model was working, the obvious next step was making it cheap. Reranking 100 candidates with a sliding window of 4 (stride 2) is ~49 windows per query; on H100 each window takes ~1.5 seconds end-to-end for a typical V3 split. We tried two families of speedups borrowed from text reranking. Both were losses worth describing. One reconciliation note, since several latency figures appear in this post: 921.8 ms is GPU-only time for a single window, from CUDA events in the profile section. The ~1.5 s per window here is wall-clock: GPU time plus ~150 ms of CPU image preprocessing plus host overhead, i.e. the 76.6 s/query smoke-run wall time divided by 49 windows. The 64–77 s/query numbers on smoke splits are measured wall-clock, which moves with each split's page-image sizes. Same checkpoint, same resolution throughout.

Tournament scheduling

In text reranking, tournament scheduling is the standard alternative to a sliding window: group the 100 candidates into windows of 4, promote the top-K from each window, and repeat until one window remains. The accounting deserves honesty up front. With PROMOTE_K=2 the pool shrinks 100 → 50 → 26 → 14 → 8 → 4 in six rounds, but the total window count is 25 + 13 + 7 + 4 + 2 + 1 = 52, essentially the same as the 49 sliding windows. As configured, this scheme was never going to be much faster; the 1.2× we measured comes from scheduling differences, not fewer forward passes. The variant that actually cuts compute is PROMOTE_K=1 (25 + 7 + 2 + 1 = 35 windows), at the price of much more aggressive pruning. We ran the gentler K=2 variant first to check quality before chasing the speedup. On the smoke split (computer_science_en, 5 queries, note that the absolute number is on the 5-query subset, not the full 215-query evaluation in the leaderboard table above):

We hypothesize that the reason tournament scheduling works on text and not on vision is error rates in the early rounds. Tournament pruning is unforgiving: once a document loses an early round, it never comes back. A strong text reranker makes few early-round mistakes, so the right documents survive to the final. On hard visual candidates from a strong first-stage like ColQwen2.5, early rounds are much noisier: two visually similar but semantically distinct pages can swap ranks from one round to the next, and every such swap that goes the wrong way permanently knocks a relevant page out of the pool. The full sliding window keeps every document in play longer and lets mis-ranked ones bubble back up.

That quality result also answers the PROMOTE_K=1 question without running it: if the gentler K=2 pruning already discards documents it can never recover, the variant that would actually deliver a speedup prunes twice as hard. 5 queries is small; we stopped here because the drop was too large to be a tuning issue.

Tournament vs sliding window: quality and latency side-by-side

First-token readout

The model generates [X] > [X] > [X] > [X], which works out to 18 decoded tokens per window on the 2B. Text reranking has a standard trick for this: first-token readout (FIRST, Reddy et al. 2024). Instead of decoding the full permutation, you read the logits of the very first generated position, where the distribution over the candidate-identifier tokens already induces a complete ordering of the window: rank the candidates by their identifier's logit and stop. Those logits come free with the prefill, so the decode loop is skipped entirely.

Two things put it off the table here. The first is the ceiling. For a text reranker, prompts are cheap to prefill and the sequential decode dominates per-window latency, so skipping it is a large multiplier. In our profile the balance is inverted: the ViT encoder and prefill together are 62% of window GPU time, and the decode loop is only 38% (~348 ms of 922 ms). Even deleting the decode entirely buys at most ~1.6× per window; the image encoding bill is due regardless of how you read the answer. The second is training: FIRST models are fine-tuned with a ranking loss on the first-position logits, while ours only ever saw cross-entropy on full permutations. The pointwise-readout experiment above suggests our model's probability mass on early tokens is reasonable but not concentrated, so an uncalibrated first-token readout is an open quality eval that we did not run once a bigger, free win appeared in the next section.

We expect it to be partially recoverable with a short fine-tune or calibration on the first-position logits; at a 1.6× ceiling we didn't pursue it.

---

Where multimodal inference actually spends its time

This is the section that surprised us most. We expected decoding to be the bottleneck, and for listwise text rerankers that expectation is well founded: prompts are cheap to prefill and the sequential decode of the ranking string dominates latency, which is exactly why tricks like first-token readout exist. With four document page images in every window, the balance flips.

We instrumented one window end-to-end. With four V3 page images per window at the default resolution, on the 2B listwise model:

The decoding cost is real, but the single biggest operation is the ViT image encoder, at nearly half of every window. Median visual tokens per window: 58,512 ViT patches per window (14,628 LLM tokens after 2×2 merge; ~3.7k per image). Decoding all 18 tokens of the permutation costs less GPU time than encoding the four input images.

A relevant detail from the training recipe matters here. The model was trained with image_size: 512: each page resized to 512×512 (≈262K pixels), yielding 256 image tokens per page after Qwen3.5's patch+merge, so four pages fit comfortably in the 2,048-token vision budget. The profiles above are measured at the processor's inference default (max_pixels = 16,777,216, ≈4096×4096), which lets each V3 page through essentially uncapped at ~14.6k ViT patches (~3.7k tokens after merge), roughly 14× the training resolution. To see what that resolution gap costs and buys, we swept three inference settings across all 16 V3 splits:

Two things are true at once. Higher inference resolution helps, even far above the training operating point: the model at the 16M default beats its own training resolution by +5.6. But the returns flatten hard above 1M pixels: the last 6× of ViT compute buys only +0.8 NDCG@10, within noise on most splits. 1M pixels is the operating point. At training resolution (262k) the model drops below jina-m0 and Qwen3-VL-Reranker-2B; the ViT features become too coarse for the fine-grained page discrimination V3 demands.

The gap between 262k and 1M also raises a question we have not run: training at 1M pixels instead of 512². Nothing blocks it (the backbone's context window is 256k tokens, and inference already feeds it 14.6k visual tokens per window); it just multiplies the training sequence length by 3×, with the memory and step-time cost that implies. Given how small the training run is, it is a natural next experiment. One happy accident of the budget-driven 512² choice: it shows you can train a visual reranker at 256 tokens per page and recover the quality at inference time by raising resolution, rather than paying for high-resolution training.

The same profile, across the whole grid

The breakdown above is anchored on the 2B listwise model. The picture changes when you change either knob, in ways that are not obvious from the headline NDCG numbers. We ran the same CUDA-event instrumentation against all six checkpoints from the size×recipe grid. The relevant snapshots:

PW=pointwise and LW=listwise

Three things shift across the grid:

• The ViT cost is flat within the Qwen3.5 2B/4B vision tower (427 ms/call) and roughly halved at 0.8B (~175 ms/call). The 0.8B ships a 12-layer encoder instead of the 24-layer one shared by 2B and 4B. So "shrink the model to save on the ViT" works only when you change vision-tower spec; going 2B → 4B buys nothing on the encoder.

• Per-token decode is flat from 0.8B to 2B (~20 ms/tok), then jumps at 4B (~28 ms/tok). On H100 the decode loop is memory-bound; the Qwen3.5 hybrid Gated DeltaNet+attention decoder simply doesn't move enough bytes per step at 0.8B/2B to differentiate. The 4B tier widens hidden state and adds layers, which is when per-token decode finally rises. The architectural payoff of the hybrid: shrinking to 0.8B buys you fewer ViT FLOPs and faster prefill, but not faster per-token decode.

• What dominates per-call cost flips with both knobs. Pointwise is always ViT-dominated (59–76% of the call) because there is no decode loop. 2B listwise is the ViT/decode tie the headline table shows. 0.8B listwise and 4B listwise are both decode-dominated (53–56%): halving the ViT (0.8B) exposes the decode loop, and scaling the per-token cost up (4B) does the same. So the "ViT eats half the wall clock" framing earlier is recipe- and size-specific; it's tightly true at the (2B, listwise) operating point this post is anchored on, but it's not a general law of multimodal rerankers.

Projecting per-call cost out to per-query at the rerank-100 protocol (25 calls/query for pointwise, 49 windows/query for listwise sliding), and plotting it against quality:

The cost spread across the grid is roughly 12× from 0.8B pointwise (6.5 s/query) to 4B listwise (77 s/query). Quality spans 48.2 → 64.7 over the same range. Production batching cuts the listwise per-query numbers by 4–8× in practice without changing the relative ordering; we report single-call numbers here so they line up with the per-call profile.

What this means in practice

If you only care about latency, the honest answer is not the cheapest cell in the grid. 0.8B pointwise runs at 6.5 s per query at rerank-100, but at 48.2 V3 @10 it lands below the first-stage on several splits, so you would get better rankings, faster, by not reranking at all. That row is a cautionary data point, not an operating point. The real latency floor among cells that earn their keep is 2B pointwise.

Counting wall time redraws the frontier we would have picked from quality alone. On the quality-versus-cost scatter in the profile section (per-query wall time at rerank-100 against V3 NDCG@10), the time-quality frontier runs: 0.8B pointwise (cheap but broken), 2B pointwise (14.1 s, 59.87, the fastest cell that clearly beats first-stage), 2B listwise (45.7 s, 62.66, the model this post is anchored on), and 4B listwise (77.4 s, 64.69, the highest quality we hit and the slowest by a long way; the profile section explains why). The other two cells are dominated. 2B pointwise beats 0.8B listwise on both axes at once: half the wall time and +1.6 NDCG. It also effectively retires 4B pointwise, which costs 4 s more per query for the same quality.

Two footnotes to that frontier. First, if your binding constraint is accelerator memory rather than latency, 0.8B listwise comes back into play: it matches the 2B-class open rerankers on V3 from a model roughly 2.5× smaller. It loses on wall time because 49 decode-heavy windows outweigh the smaller ViT, not because the quality isn't there. Second, 2B pointwise earns its frontier spot on speed, not headline quality: it adds +4.3 over first-stage, more than half the 7.1 that 2B listwise buys for 3.2× the time. So pick by whichever axis binds: latency floor, 2B pointwise; quality ceiling, 4B listwise; balanced default, 2B listwise.

The cheaper lever: rerank fewer candidates

Resolution and backbone-size choices help with per-window cost, but the bigger knob is, quite simply, how many candidates you rerank at all. The sliding window over 100 candidates takes 49 windows per query. Most of those windows are spent re-ordering candidates deep in the pool that will never reach the top 10 anyway.

We measured what happens when you simply truncate to the top-K candidates by first-stage retrieval score and run the same sliding window on the shorter list. Full evaluation, all 16 EN+FR splits, same protocol as the leaderboard (ColQwen2.5 first-stage, retrieve top-100):

The last column is the hard ceiling: reranking K candidates cannot surface a document the first stage ranked below K, so first-stage recall@K bounds what any reranker can do from there. That makes the trade explicit. Truncating to 20 candidates cuts the window count 5.4× and keeps 85% of the rerank lift. Truncating to 10 cuts it 12× and keeps 58%. Reranking never falls below the first-stage at any depth on any split.

The mean hides a spread that depends on your first stage. At K=10 the per-split cost ranges from −1.3% (computer_science_en, where first-stage recall@10 is 81.0) to −10.2% (finance_fr_en, recall@10 46.0). The correlation is directional rather than exact (physics forfeits the most recall yet loses the least NDCG, because its deep candidates were not reaching the top-10 even with the full pool), but the guidance is clear: the better your first stage, the cheaper the truncation.

For production, this means: don't switch scoring methods to go faster; shrink the candidate pool, and pick the depth from your own first-stage recall curve. With a strong retriever, K=20 is close to free (−1.7% mean here) at 5× fewer windows; K=10 is 12× fewer windows but gives back nearly half the rerank lift on average, and much more where first-stage recall is weak. The model stays in-distribution and the cross-document attention stays intact either way; what you give up is exactly the documents your first stage buried below K.

---

Recommendations, if you're building something similar

• Pin your protocol before you measure anything. A surprising fraction of "what reranker is best" arguments boil down to comparing rerank-10 to rerank-100, or two different first-stages, or two different image-prompt templates. Decide on (depth, first-stage, metric, window scheme, scoring template) and fix it.
• For a 2B-class multimodal reranker, go generative listwise, not pointwise. Across 0.8B / 2B / 4B the listwise recipe wins on V3 at every size, and the lift is largest where the model is smallest. Pointwise saturates with scale; listwise keeps benefiting. Reach for pointwise only if per-call latency is the hard constraint and you can live with a 3.0–10.0 NDCG@10 deficit.
• Don't read a listwise model pointwise at deploy. The strength of a listwise reranker is that it scores documents relative to each other. Trying to escape the cost of listwise by scoring documents one at a time (pointwise readout) throws away exactly the signal you trained for; on the same checkpoint it cost us roughly three quarters of the rerank lift on our smoke split.
• Profile your forward pass on your operating point, not in the abstract. At 2B listwise, ViT and decode each carry ~40% of the per-window time. The same recipe at 0.8B is decode-dominated; at 4B listwise also decode-dominated; pointwise at every size is ViT-dominated. "Decoding is slow" and "ViT is slow" are both true sometimes, and the right answer depends on which cell you're in.
• Shrink the candidate pool, not the model. A strong first stage makes truncation cheap: on ViDoRe V3, reranking the top-20 kept 85% of the rerank lift at 5× fewer windows, and top-10 kept 58% at 12× fewer. The cost per split tracked first-stage recall@K (−1.3% where recall@10 was 81.0, −10.2% where it was 46.0). The model stays in-distribution and cross-doc attention stays intact; check your first-stage recall curve before picking a depth.

---

Citation

@misc{ananya2026lightonrerank,
  title={One Adapter, Both Modalities: Field Notes from Building and Serving a Multimodal Reranker},
  author={Ananya, Ishrat Jahan and Chatelain, Amelie},
  year={2026},
  howpublished={\url{https://huggingface.co/blog/lightonai/lighton-rerank}},
}

---

Acknowledgements

We thank Antoine Chaffin for his valuable feedback on this post. We are grateful to the teams behind the ViDoRe benchmark and the BEIR benchmark, and to the open-source retrieval community, in particular the authors of Nomic Embed Multimodal for the mined vision negatives and the Qwen team for the Qwen3.5 backbones. This project is also supported by the OpenEuroLLM project, co-funded by the Digital Europe Programme under GA no. 101195233. For more information see https://openeurollm.eu.

---

Bibliography & further reading

LightOn lineage

• lightonai/MonoQwen2-VL-v0.1 — previous-generation vision reranker, the starting point for this work.

Listwise reranking with LLMs

• Sun et al., 2023. Is ChatGPT Good at Search? Investigating LLMs as Re-Ranking Agents. RankGPT, sliding window. arXiv:2304.09542
• Pradeep et al., 2023. RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze! arXiv:2312.02724
• Cai, 2026. When Vision Meets Texts in Listwise Reranking. arXiv:2601.20623
• Li et al. 2025, ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Reranking., arXiv:2506.03487

ViDoRe and friends

• Faysse et al., 2024. ColPali: Efficient Document Retrieval with Vision Language Models. arXiv:2407.01449
• Loisin et al., 2026, ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios arXiv:2601.08620
• vidore/colqwen2.5-v0.2 — first-stage retriever used in all V3 numbers here.

Other rerankers we benchmarked

• jinaai/jina-reranker-m0 — Qwen2-VL-2B + MLP, off-the-shelf. Caveat: needs the model's official compute_score(doc_type="image") template; the wrong vision prompt costs 18–30 NDCG@10 points (we tripped on this once).
• Qwen/Qwen3-VL-Reranker-2B, Qwen/Qwen3-VL-Reranker-8B — off-the-shelf. The 2B is a thinking model; score via the vLLM score() pooling head, not generate() + first-token (the model emits <think> etc. first and the first-token logits are not the rerank signal).

---

Appendix

Appendix: Hyperparameters

Vision loss weight sweep: We swept vw ∈ {1.0, 1.3, 1.8} × W ∈ {3, 4} at 100 steps. 1.0 and 1.3 are equivalent on both modalities; 1.8 collapses vision accuracy to zero while text stays saturated. The failure mode is over-weighting vision gradients, not under-weighting text.

Appendix: ViDoRe V3 full results

NDCG@10. First-stage: ColQwen2.5, retrieve_k=100, rerank_k=100 (full pool). Listwise: sliding window W=4, stride=2. Pointwise: batched, bs=4. Images at the processor default resolution.

Appendix: BEIR text reranking results

NDCG@10. First-stage: BM25, retrieve_k=100, rerank_k=100. Listwise: sliding window W=4, stride=2.

Footnote: "⚠️" = trained on this dataset (NQ, MSMARCO).