Ecommerce Embedding Model Benchmarks

This Space contains benchmark results conducted as part of the release of our ecommerce embedding models: Marqo-Ecommerce-L and Marqo-Ecommerce-B.

Our benchmarking process was divided into two distinct regimes, each using different datasets of ecommerce product listings: marqo-ecommerce-hard and marqo-ecommerce-easy. Both datasets contained product images and text and only differed in size. The "easy" dataset is approximately 10-30 times smaller (200k vs 4M products), and designed to accommodate rate-limited models, specifically Cohere-Embeddings-v3 and GCP-Vertex (with limits of 0.66 rps and 2 rps respectively). The "hard" dataset represents the true challenge, since it contains four million ecommerce product listings and is more representative of real-world ecommerce search scenarios.

Within both these scenarios, the models were benchmarked against three different tasks:

Google Shopping Text-to-Image

Google Shopping Category-to-Image

Amazon Products Text-to-Image

As part of this launch, we also released two evaluation datasets: Marqo/google-shopping-general-eval and Marqo/amazon-products-eval.

For more information on these models, benchmark results, and how you can run these evaluations yourself, visit our blog post.

Marqo-Ecommerce-Hard

Google Shopping Text to Image 1m

Embedding Model	mAP	R@10	MRR	nDCG@10
Amazon-Titan-MultiModal	0.682	0.878	0.683	0.726

Embedding Model	mAP	R@10	MRR	nDCG@10
Marqo-Ecommerce-L	0.682	0.878	0.683	0.726
Marqo-Ecommerce-B	0.623	0.832	0.624	0.668
ViT-SO400M-14-SigLip	0.573	0.763	0.574	0.613
ViT-L-16-SigLip	0.54	0.722	0.54	0.577
ViT-B-16-SigLip	0.476	0.66	0.477	0.513
Amazon-Titan-MultiModal	0.475	0.648	0.475	0.509
Jina-V1-CLIP	0.285	0.402	0.285	0.306

Google Shopping Category to Image 1m

Embedding Model	mAP	P@10	MRR	nDCG@10
Amazon-Titan-MultiModal	0.463	0.652	0.822	0.666

Embedding Model	mAP	P@10	MRR	nDCG@10
Marqo-Ecommerce-L	0.463	0.652	0.822	0.666
Marqo-Ecommerce-B	0.423	0.629	0.81	0.644
ViT-SO400M-14-SigLip	0.352	0.516	0.707	0.529
ViT-L-16-SigLip	0.324	0.497	0.687	0.509
ViT-B-16-SigLip	0.277	0.458	0.66	0.473
Amazon-Titan-MultiModal	0.246	0.429	0.642	0.446
Jina-V1-CLIP	0.123	0.275	0.504	0.294

Amazon Products Text to Image 3m

Embedding Model	mAP	R@10	MRR	nDCG@10
Amazon-Titan-MultiModal	0.658	0.854	0.663	0.703

Embedding Model	mAP	R@10	MRR	nDCG@10
Marqo-Ecommerce-L	0.658	0.854	0.663	0.703
Marqo-Ecommerce-B	0.592	0.795	0.597	0.637
ViT-SO400M-14-SigLip	0.56	0.742	0.564	0.599
ViT-L-16-SigLip	0.544	0.715	0.548	0.58
ViT-B-16-SigLip	0.48	0.65	0.484	0.515
Amazon-Titan-MultiModal	0.456	0.627	0.457	0.491
Jina-V1-CLIP	0.265	0.378	0.266	0.285

Marqo-Ecommerce-Easy

Google Shopping Text to Image

Embedding Model	mAP	R@10	MRR	nDCG@10
Amazon-Titan-MultiModal	0.879	0.971	0.879	0.901

Embedding Model	mAP	R@10	MRR	nDCG@10
Marqo-Ecommerce-L	0.879	0.971	0.879	0.901
Marqo-Ecommerce-B	0.842	0.961	0.842	0.871
ViT-SO400M-14-SigLip	0.792	0.935	0.792	0.825
ViT-L-16-SigLip	0.754	0.907	0.754	0.789
GCP-Vertex	0.74	0.91	0.74	0.779
ViT-B-16-SigLip	0.701	0.87	0.701	0.739
Amazon-Titan-MultiModal	0.694	0.868	0.693	0.733
Jina-V1-CLIP	0.48	0.638	0.48	0.511
Cohere-embedding-v3	0.358	0.515	0.358	0.389

Google Shopping Category to Image

Embedding Model	mAP	P@10	MRR	nDCG@10
Amazon-Titan-MultiModal	0.515	0.358	0.764	0.558

Embedding Model	mAP	P@10	MRR	nDCG@10
Marqo-Ecommerce-L	0.515	0.358	0.764	0.59
Marqo-Ecommerce-B	0.479	0.336	0.744	0.558
ViT-SO400M-14-SigLip	0.423	0.302	0.644	0.487
GCP-Vertex	0.417	0.298	0.636	0.481
ViT-L-16-SigLip	0.392	0.281	0.627	0.458
ViT-B-16-SigLip	0.347	0.252	0.594	0.414
Amazon-Titan-MultiModal	0.308	0.231	0.558	0.377
Jina-V1-CLIP	0.175	0.122	0.369	0.229
Cohere-embedding-v3	0.136	0.11	0.315	0.178

Amazon Products Text to Image

Embedding Model	mAP	R@10	MRR	nDCG@10
Amazon-Titan-MultiModal	0.928	0.978	0.928	0.914

Embedding Model	mAP	R@10	MRR	nDCG@10
Marqo-Ecommerce-L	0.928	0.978	0.928	0.94
Marqo-Ecommerce-B	0.897	0.967	0.897	0.914
ViT-SO400M-14-SigLip	0.86	0.954	0.86	0.882
ViT-L-16-SigLip	0.842	0.94	0.842	0.865
GCP-Vertex	0.808	0.933	0.808	0.837
ViT-B-16-SigLip	0.797	0.917	0.797	0.825
Amazon-Titan-MultiModal	0.762	0.889	0.763	0.791
Jina-V1-CLIP	0.53	0.699	0.53	0.565
Cohere-embedding-v3	0.433	0.597	0.433	0.465