lemonhat/Qwen2.5-Coder-7B-Instruct-t1_100k_v3_tag5_cleaned_hermes

t1_100k_v3_tag5_cleaned_hermes

This model is a fine-tuned version of Qwen/Qwen2.5-Coder-7B-Instruct on the t1_100k_v3_tag5_cleaned_hermes dataset. It achieves the following results on the evaluation set:

• Loss: 0.2204

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• total_eval_batch_size: 4
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
0.283	0.0356	100	0.3038
0.2702	0.0712	200	0.2718
0.2706	0.1068	300	0.2616
0.2664	0.1425	400	0.2548
0.2177	0.1781	500	0.2500
0.2401	0.2137	600	0.2458
0.2687	0.2493	700	0.2453
0.2346	0.2849	800	0.2421
0.2343	0.3205	900	0.2386
0.2171	0.3561	1000	0.2365
0.2435	0.3917	1100	0.2352
0.218	0.4274	1200	0.2335
0.2566	0.4630	1300	0.2323
0.2528	0.4986	1400	0.2308
0.21	0.5342	1500	0.2293
0.2361	0.5698	1600	0.2282
0.229	0.6054	1700	0.2268
0.221	0.6410	1800	0.2254
0.2401	0.6766	1900	0.2245
0.2094	0.7123	2000	0.2233
0.2115	0.7479	2100	0.2221
0.2026	0.7835	2200	0.2219
0.2364	0.8191	2300	0.2214
0.2317	0.8547	2400	0.2208
0.2098	0.8903	2500	0.2207
0.2145	0.9259	2600	0.2207
0.2144	0.9615	2700	0.2206
0.2393	0.9972	2800	0.2204

Framework versions

• Transformers 4.46.1
• Pytorch 2.6.0+cu124
• Datasets 3.1.0
• Tokenizers 0.20.3