T2V-CompBench Leaderboard

🏆 Welcome to the leaderboard of the T2V-CompBench! 🎦 A Comprehensive Benchmark for Compositional Text-to-video Generation

1400 Prompts: We analyze 1.67 million real-user prompts to extract high-frequency nouns, verbs, and adjectives, resulting in a suite of 1,400 prompts.
7 Compositional Categories: We evaluate multiple-object compositionality on attributes, actions, interactions, quantities, and spatio-temporal dynamics, covering 7 categories.
Evaluation metrics: We design MLLM-based, Detection-based, and Tracking-based evaluation metrics for compositional T2V generation, all validated by human evaluations.
Valuable Insights: We provide insightful analysis on current models' ability, highlighting the significant challenge of compositional T2V generation.

Join Leaderboard: Please follow the steps in our github repository to prepare the videos and run the evaluation scripts. Before uploading the generated .csv files here, please conduct a final check by carefully reading this instruction. After clicking the Submit Eval! button, click the Refresh button. Then, you can successfully showcase your model's performance on our leaderboard!

Model Information: What are the details of these Video Generation Models? See Appendix B of our paper. We will provide more details soon.

Evaluated by T2V-CompBench Team (Uncheck to view all submissions)

Model Name (clickable)	Evaluated by	Date	Total Avg. Score	Selected Avg. Score	Consistent Attribute Binding	Dynamic Attribute Binding	Spatial Relationships	Motion Binding	Action Binding	Object Interactions	Generative Numeracy
ModelScope	T2V-CompBench Team	2024-12-01	0.5661	0.5661	0.6931	0.0624	0.5979	0.2867	0.8722	0.8309	0.6066

Model Name (clickable)	Evaluated by	Date	Total Avg. Score	Selected Avg. Score	Consistent Attribute Binding	Dynamic Attribute Binding	Spatial Relationships	Motion Binding	Action Binding	Object Interactions	Generative Numeracy
PixVerse V3	T2V-CompBench Team	2024-12-01	0.5661	0.5661	0.706	0.0624	0.5979	0.2867	0.8722	0.8309	0.6066
Kling 1.0	T2V-CompBench Team	2024-12-01	0.4658	0.4658	0.6931	0.0098	0.569	0.2562	0.5787	0.7128	0.4413
Dreamina 1.2	T2V-CompBench Team	2024-12-01	0.4604	0.4604	0.6913	0.0051	0.5773	0.2361	0.5924	0.6824	0.438
T2V-Turbo-V2	T2V-CompBench Team	2024-12-01	0.4317	0.4317	0.6723	0.0127	0.5025	0.2556	0.6087	0.6439	0.3261
CogVideoX-5B	T2V-CompBench Team	2024-12-01	0.4189	0.4189	0.6164	0.0219	0.5172	0.2658	0.5333	0.6069	0.3706
VideoTetris	T2V-CompBench Team	2024-12-01	0.4054	0.4054	0.6211	0.0104	0.4832	0.2249	0.4939	0.6578	0.3467
VideoCrafter2	T2V-CompBench Team	2024-12-01	0.4015	0.4015	0.6182	0.0103	0.4838	0.2259	0.503	0.6365	0.333
Gen-3	T2V-CompBench Team	2024-12-01	0.4009	0.4009	0.598	0.0687	0.5194	0.2754	0.5233	0.5906	0.2306
Vico	T2V-CompBench Team	2024-12-01	0.3926	0.3926	0.5887	0.0107	0.4974	0.2219	0.5111	0.5957	0.323
Open-Sora 1.2	T2V-CompBench Team	2024-12-01	0.385	0.385	0.5639	0.0189	0.5063	0.2468	0.4833	0.5039	0.3719
Mochi	T2V-CompBench Team	2024-12-01	0.3842	0.3842	0.5973	0.0246	0.548	0.2334	0.4759	0.5381	0.2718
Gen-2	T2V-CompBench Team	2024-12-01	0.3828	0.3828	0.5795	0.0109	0.5126	0.2173	0.4413	0.6144	0.3039
Open-Sora 1.1	T2V-CompBench Team	2024-12-01	0.3722	0.3722	0.5414	0.0109	0.5406	0.2261	0.5037	0.5565	0.2259
Pika 1.0	T2V-CompBench Team	2024-12-01	0.3695	0.3695	0.5536	0.0128	0.465	0.2234	0.425	0.5198	0.387
Show-1	T2V-CompBench Team	2024-12-01	0.369	0.369	0.567	0.0115	0.4544	0.2291	0.3881	0.6244	0.3086
Open-Sora-Plan v1.3.0	T2V-CompBench Team	2024-12-01	0.367	0.367	0.6076	0.0119	0.5162	0.2377	0.4524	0.4483	0.2952
LVD	T2V-CompBench Team	2024-12-01	0.3398	0.3398	0.5439	0.0171	0.5405	0.2457	0.3802	0.4502	0.2008
ModelScope	T2V-CompBench Team	2024-12-01	0.3153	0.3153	0.5148	0.0161	0.4118	0.2408	0.3639	0.4613	0.1986
Latte	T2V-CompBench Team	2024-12-01	0.3129	0.3129	0.4713	0.008	0.434	0.2155	0.4146	0.4146	0.232
Open-Sora-Plan v1.0.0	T2V-CompBench Team	2024-12-01	0.307	0.307	0.4246	0.0086	0.452	0.2148	0.4009	0.415	0.2331
ZeroScope	T2V-CompBench Team	2024-12-01	0.3015	0.3015	0.4011	0.0091	0.4287	0.2454	0.3661	0.4196	0.2408
AnimateDiff	T2V-CompBench Team	2024-12-01	0.2736	0.2736	0.4325	0.0097	0.392	0.2227	0.2844	0.397	0.1767
MagicTime	T2V-CompBench Team	2024-12-01	0.0151	0.0151		0.0151

Model Name (clickable)	Evaluated by	Date	Consistent Attribute Binding-Color	Consistent Attribute Binding-Shape	Consistent Attribute Binding-Texture	2D Spatial Relationships-Coexist	2D Spatial Relationships-Acc.	2D Spatial Relationships-Acc.Score	Motion Binding-Motion Level	Motion Binding-Acc.	Action Binding-Common	Action Binding-Uncommon	Object Interactions-Physical	Object Interactions-Social	Total Avg. Score	Selected Avg. Score
PixVerse V3	T2V-CompBench Team	2024-12-01	0.7898	0.5768	0.6381	89%	71%	0.8231	28.09	38%	0.9016	0.7546	0.8219	0.84	0.5661	0.5661
Kling 1.0	T2V-CompBench Team	2024-12-01	0.7971	0.4673	0.6149	90%	57%	0.8634	17.27	35%	0.5863	0.5481	0.653	0.7726	0.4658	0.4658
Dreamina 1.2	T2V-CompBench Team	2024-12-01	0.7798	0.5601	0.5738	90%	62%	0.8777	12.71	23%	0.6225	0.4722	0.6411	0.7237	0.4604	0.4604
T2V-Turbo-V2	T2V-CompBench Team	2024-12-01	0.7781	0.5351	0.5452	77%	54%	0.7967	19.14	37%	0.6694	0.3657	0.5907	0.697	0.4317	0.4317
CogVideoX-5B	T2V-CompBench Team	2024-12-01	0.7202	0.4333	0.5167	71%	61%	0.8706	23.33	34%	0.5824	0.337	0.5607	0.653	0.4189	0.4189
VideoTetris	T2V-CompBench Team	2024-12-01	0.6957	0.4667	0.5589	74%	53%	0.7831	9.94	21%	0.541	0.3056	0.62	0.6956	0.4054	0.4054
VideoCrafter2	T2V-CompBench Team	2024-12-01	0.6717	0.4792	0.578	73%	59%	0.7914	8.96	24%	0.5472	0.3259	0.5607	0.7122	0.4015	0.4015
Gen-3	T2V-CompBench Team	2024-12-01	0.6652	0.5077	0.528	68%	67%	0.8728	22.59	42%	0.5639	0.3611	0.5663	0.6148	0.4009	0.4009
Vico	T2V-CompBench Team	2024-12-01	0.6705	0.4476	0.4619	80%	50%	0.8083	9.12	20%	0.5609	0.312	0.5263	0.6652	0.3926	0.3926
Open-Sora 1.2	T2V-CompBench Team	2024-12-01	0.6038	0.4321	0.5607	76%	50%	0.8905	15.98	30%	0.5069	0.3889	0.4396	0.5681	0.385	0.385
Mochi	T2V-CompBench Team	2024-12-01	0.7043	0.4393	0.4774	78%	75%	0.8045	18.55	34%	0.5336	0.2454	0.493	0.5833	0.3842	0.3842
Gen-2	T2V-CompBench Team	2024-12-01	0.6721	0.3994	0.5131	83%	53%	0.8162	7.73	20%	0.4894	0.2491	0.5681	0.6607	0.3828	0.3828
Open-Sora 1.1	T2V-CompBench Team	2024-12-01	0.5669	0.422	0.5304	77%	69%	0.8323	10.87	25%	0.5613	0.2731	0.507	0.6059	0.3722	0.3722
Pika 1.0	T2V-CompBench Team	2024-12-01	0.6095	0.4315	0.5119	72%	55%	0.7841	8.13	17%	0.465	0.2648	0.4348	0.6048	0.3695	0.3695
Show-1	T2V-CompBench Team	2024-12-01	0.5964	0.4774	0.5601	62%	56%	0.8387	11.74	26%	0.4331	0.2083	0.5659	0.683	0.369	0.369
Open-Sora-Plan v1.3.0	T2V-CompBench Team	2024-12-01	0.7107	0.4244	0.5429	76%	59%	0.8726	15.89	22%	0.4938	0.287	0.4393	0.4574	0.367	0.367
LVD	T2V-CompBench Team	2024-12-01	0.6155	0.4139	0.4458	63%	90%	0.9054	13.73	47%	0.412	0.2528	0.3752	0.5252	0.3398	0.3398
ModelScope	T2V-CompBench Team	2024-12-01	0.5826	0.344	0.456	62%	44%	0.8011	16.98	32%	0.4025	0.2093	0.3815	0.5411	0.3153	0.3153
Latte	T2V-CompBench Team	2024-12-01	0.4657	0.3613	0.528	61%	51%	0.7856	12.76	14%	0.4623	0.2241	0.3904	0.4389	0.3129	0.3129
Open-Sora-Plan v1.0.0	T2V-CompBench Team	2024-12-01	0.4357	0.3887	0.4107	66%	58%	0.732	7.54	17%	0.4583	0.1713	0.4067	0.4233	0.307	0.307
ZeroScope	T2V-CompBench Team	2024-12-01	0.4829	0.1905	0.3012	50%	65%	0.7836	21	29%	0.4042	0.2139	0.297	0.5422	0.3015	0.3015
AnimateDiff	T2V-CompBench Team	2024-12-01	0.4619	0.3113	0.4196	50%	52%	0.818	12.79	21%	0.3053	0.2009	0.3604	0.4337	0.2736	0.2736
MagicTime	T2V-CompBench Team	2024-12-01													0.0151	0.0151

T2V-CompBench Leaderboard

Submit on T2V-CompBench Introduction

📮

✉️✨ Submit your model evaluation CSV files here!