Open source evaluation framework: hallucination + safety + accuracy + cost
Hi WhiteCircle team 👋
Safety and accuracy tradeoffs in LLM guards is exactly the right problem to benchmark. For teams choosing LLM guardrails for production, cost is the third dimension that determines what actually ships.
I built an open source LLM Evaluation Framework that measures:
→ 🔍 Hallucination Rate — safety-relevant overconfident outputs
→ 🎯 Accuracy — precision on the task
→ 💰 Cost per 1K tokens — guardrail inference cost at production scale
→ ⚡ Latency p95 — guardrails in the critical path have strict latency budgets
→ 🧠 Reasoning Quality — for guard models that explain their safety decisions
Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Would love to discuss how accuracy/cost/safety measurement applies to guardrail benchmarking!