mental_health_Mistral_7B_Instruct_v0.3

This is a LoRA fine-tuned adapter for mistralai/Mistral-7B-Instruct-v0.3, trained on the Amod/mental_health_counseling_conversations dataset.
The goal of this model is to enhance the base Mistral model with improved empathy and contextual understanding in mental health-related conversations.

---

Model description

This adapter was trained using PEFT with LoRA on top of a 4-bit quantized Mistral 7B model using bitsandbytes.
The adapter enables the base model to produce more emotionally supportive and human-like responses in mental health conversations.
It is lightweight (~100MB) and optimized for efficient deployment.

⚠️ This model is not intended for clinical use. It is a research and educational tool only.

---

Training and evaluation data

The model was trained on the Amod/mental_health_counseling_conversations dataset, containing dialog pairs between users and licensed therapists.
These conversations span a range of psychological topics including anxiety, depression, self-esteem, and motivation.

---

Training hyperparameters

The following hyperparameters were used:

• learning_rate: 1e-5
• train_batch_size: 8
• eval_batch_size: 8
• weight_decay: 0.01
• optimizer: Adam (betas=(0.9, 0.999), eps=1e-8)
• lr_scheduler_type: cosine
• warmup_ratio: 0.05
• num_epochs: 2
• fp16: True
• save_steps: 500
• save_total_limit: 2
• logging_steps: 50
• report_to: none
• quantization: 4-bit using bitsandbytes (NF4 + double quantization)

---

Training results

Training Loss	Epoch	Validation Loss	Eval Runtime (s)	Samples/sec	Steps/sec
1.7287	2.0	1.8093	204.13	1.724	0.216

---

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model = "mistralai/Mistral-7B-Instruct-v0.3"
adapter = "jpate190/mental_health_Mistral_7B_Instruct_v0.3"

# Load tokenizer and base model
tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    load_in_4bit=True,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

# Load LoRA adapter
model = PeftModel.from_pretrained(model, adapter)

# Inference
model.eval()
messages = [
    {"role": "user", "content": "Hey, I've been feeling really anxious and unmotivated lately. Any advice?"}
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
output = model.generate(**inputs, max_new_tokens=512, do_sample=True, pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Framework versions

PEFT 0.7.1

Transformers 4.36.1

PyTorch 2.0.0

Datasets 2.1.0

Tokenizers 0.15.0