---
language:
- en
license: llama2
tags:
- code
- llama2
- full-fine-tuning
- mask-fine-tuning
- coding
datasets:
- tulu3_persona_python
- evol_code
- code_alpaca
base_model: meta-llama/Llama-2-7b-hf
---

# llama2-7b-coding-fft

This model is a **Full Fine-Tuned (FFT)** version of LLaMA2-7B on coding datasets, trained as part of replicating the [Mask Fine-Tuning (MFT) paper](https://arxiv.org/abs/2503.22764v1).

## Model Details

- **Base Model:** meta-llama/Llama-2-7b-hf
- **Training Type:** Full Fine-Tuning (FFT)
- **Domain:** Coding
- **Hardware:** TPU v4-8
- **Training Framework:** PyTorch + torch_xla

## Training Data

The model was trained on 30,000 samples from three coding datasets (matching the paper):
- **Tulu 3 Persona Python:** 10,000 samples
- **Evol CodeAlpaca:** 10,000 samples
- **Code-Alpaca:** 10,000 samples

## Training Configuration

- **Epochs:** 2
- **Sequence Length:** 4096
- **Learning Rate:** 2e-5
- **Batch Size:** 8 (effective)
- **Optimizer:** AdamW
- **LR Scheduler:** Linear with warmup
- **Mixed Precision:** bfloat16

## Training Results

- **Final Loss:** 0.15353151041666666
- **Final Perplexity:** 1.1673020833333334
- **Training Time:** ~7 hours on TPU v4-8
- **Total Steps:** 7500

### Loss Progression
- Epoch 0: 0.42591484375
- Epoch 1: 0.15353151041666666

## Intended Use

This model serves as the **FFT baseline** for the Mask Fine-Tuning paper replication. It will be evaluated on:
- **HumanEval** (code generation benchmark)
- Target: Match paper's FFT baseline of 29.3%

## Evaluation

Evaluation on HumanEval is pending. Results will be updated here once available.

## Citation

If you use this model, please cite the original MFT paper:

```bibtex
@article{mft2025,
  title={Mask Fine-Tuning},
  author={[Authors from paper]},
  journal={arXiv preprint arXiv:2503.22764v1},
  year={2025}
}
```

## Reproducibility

Training configuration and code available at: [GitHub Repository](https://github.com/chrisfrancisque/mft-tpu)

## License

This model inherits the LLaMA 2 Community License from the base model.