Obito19 commited on
Commit
2e62a07
·
verified ·
1 Parent(s): 31de22b

FAQ: show how to apply the query instruction, and what omitting it costs

Browse files

The FAQ says omitting the query instruction causes "a performance degradation" but doesn't say how to apply it or how large the effect is. This adds the explicit call and a measured figure.

On 2AIRTC (12,587 native Amharic documents, 240 native topics, 7,084 human relevance judgments), harrier-oss-v1-270m @ 31de22b scores nDCG@10 0.7159 via model.encode(queries) and 0.7850 via prompt_name="web_search_query" — +0.0691, paired t-test over 240 topics, p = 4.1e-14, bootstrap 95% CI [+0.0526, +0.0860]. Documents truncated to 2,000 characters with max_seq_length=2493 so nothing is truncated; document embeddings byte-identical across conditions. On BEIR SciFact (English) the same measurement gives +0.0089.

The note about default_prompt_name is there because it's the natural thing to reach for and it doesn't work — it applies to every encode() call, so it would instrument documents too. Related: #1, which addresses the encode_query path.

One collection, one language; the 27b variant wasn't tested.

Files changed (1) hide show
  1. README.md +18 -1
README.md CHANGED
@@ -228,6 +228,23 @@ This is a way to customize text embeddings for different scenarios through natur
228
 
229
  On the other hand, there is no need to add instructions to the document side.
230
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
231
  **2. Why are my reproduced results slightly different from reported in the model card?**
232
 
233
  Different versions of `transformers` and `pytorch` could cause negligible but non-zero performance differences.
@@ -235,4 +252,4 @@ Different versions of `transformers` and `pytorch` could cause negligible but no
235
  **3. What pooling strategy does this model use?**
236
 
237
  The model uses **last-token pooling** — the embedding of the last non-padding token is used as the sentence representation.
238
- The embedding is then L2-normalized. This is handled automatically when using Sentence Transformers.
 
228
 
229
  On the other hand, there is no need to add instructions to the document side.
230
 
231
+ The instruction is not applied automatically. `model.encode(queries)` sends the query with no
232
+ instruction, and the resulting embeddings are well-formed and normalized, so an omission is not
233
+ visible from the output. Select the prompt explicitly:
234
+
235
+ ```python
236
+ model.encode(queries, prompt_name="web_search_query") # or your own task instruction
237
+ model.encode(documents) # documents take no instruction
238
+ ```
239
+
240
+ The cost of omitting it is task- and language-dependent, and can be substantial. On a native
241
+ Amharic ad-hoc retrieval collection (2AIRTC: 12,587 documents, 240 topics, 7,084 human relevance
242
+ judgments), `harrier-oss-v1-270m` scores nDCG@10 0.7159 without the instruction and 0.7850 with
243
+ `prompt_name="web_search_query"` — a difference of +0.0691 (paired t-test over 240 topics,
244
+ p = 4.1e-14). On BEIR SciFact (English) the same measurement gives +0.0089.
245
+
246
+ Note that `default_prompt_name` cannot be used to apply the instruction automatically, because it
247
+ applies to every `encode()` call including documents.
248
  **2. Why are my reproduced results slightly different from reported in the model card?**
249
 
250
  Different versions of `transformers` and `pytorch` could cause negligible but non-zero performance differences.
 
252
  **3. What pooling strategy does this model use?**
253
 
254
  The model uses **last-token pooling** — the embedding of the last non-padding token is used as the sentence representation.
255
+ The embedding is then L2-normalized. This is handled automatically when using Sentence Transformers.