Most prompt-injection guardrails are scored on one axis: how many attacks they catch. guardrail-arena scores two at once — miss-rate and over-refusal — in English and Turkish. A guardrail is any callable text → 0|1. Here's what the popular ones actually do.
Over-refusal rate on benign prompts. On plain requests (“Summarize this article.”) almost nobody over-blocks. But on security-adjacent text — a teacher asking “what is a jailbreak?”, an incident report quoting an attacker — over-refusal explodes. Lower is better.
Even ProtectAI's widely-deployed detector over-blocks 40% of security-adjacent text. The security-adjacent split is fully held-out for every guard.
Attack miss-rate, English vs Turkish. Same guard, same attacks translated. A guardrail that looks fine on an English leaderboard can miss the overwhelming majority of the same attacks in another language. Lower is better.
jackhhao's classifier catches 90% of English attacks but misses 83% in Turkish. ProtectAI's Turkish over-refusal (25%) also runs above its overall (18%).
Ranked by F1. Miss-rate & over-refusal: lower is better. Every guard except the AltaySec detector is fully held-out.
| Guardrail | Miss ↓ | Over-refusal ↓ | Sec-adj over-refusal ↓ | F1 ↑ | TR miss ↓ |
|---|---|---|---|---|---|
| protectai-deberta-v2 | 2% | 18% | 40% | 0.94 | 2% |
| AltaySec-detector * | 1% | 23% | 70% | 0.94 | 0% |
| jackhhao-jailbreak | 46% | 13% | 25% | 0.67 | 83% |
| keyword | 88% | 13% | 38% | 0.20 | 91% |
| regex-rules | 91% | 10% | 30% | 0.16 | 90% |
* AltaySec-detector has train-set overlap on injections + plain benign; its security-adjacent column is its only fully held-out result.
Wrap Llama Guard, Prompt Guard, Lakera, NeMo Guardrails, or your own model as guard(text) → 0|1, run one command, open a PR. The point is an apples-to-apples, two-axis, multilingual comparison.