Most prompt-injection guardrails are scored on one axis: how many attacks they catch. guardrail-arena scores two at once — miss-rate and over-refusal — in English and Turkish. A guardrail is any callable text → 0|1. Here's what the popular ones actually do.
Over-refusal rate on benign prompts. On plain requests (“Summarize this article.”) almost nobody over-blocks. But on security-adjacent text — a teacher asking “what is a jailbreak?”, an incident report quoting an attacker — over-refusal explodes. Lower is better.
Even ProtectAI's widely-deployed detector over-blocks 40% of security-adjacent text. The security-adjacent split is fully held-out for every guard.
Attack miss-rate, English vs Turkish. Same guard, same attacks translated. A guardrail that looks fine on an English leaderboard can miss the overwhelming majority of the same attacks in another language. Lower is better.
jackhhao's classifier catches 90% of English attacks but misses 83% in Turkish. ProtectAI's Turkish over-refusal (25%) also runs above its overall (18%).
Ranked by F1. Miss-rate & over-refusal: lower is better. Every guard except the AltaySec detector is fully held-out.
| Guardrail | Miss ↓ | Over-refusal ↓ | Sec-adj over-refusal ↓ | F1 ↑ | TR miss ↓ |
|---|
* AltaySec-detector has train-set overlap on injections + plain benign; its security-adjacent column is its only fully held-out result.
Wrap Llama Guard, Prompt Guard, Lakera, NeMo Guardrails, or your own model as guard(text) → 0|1, run one command, open a PR. The point is an apples-to-apples, two-axis, multilingual comparison.