Interpreting results

A guide for the model-picker: what each column means and how to choose.

The columns

• Robustness — 1 − ASR: the fraction of attacks the model survived. Higher is better. The CI bar is the 95% bootstrap interval — if two models’ intervals overlap, treat them as tied.
• Per-family ASR — the mini heatmap in each row. Green cells are families the model resists; red cells are where it breaks. This is the most diagnostic view: two models with equal robustness can have opposite weak spots.
• ★ Robustness / $ — robustness divided by cost per 1k decisions. The value view. Self-hosted models show ∞ (no per-call cost — but you pay in hardware and ops).
• Cost / 1k — measured dollars per thousand decisions during the benchmark run, from token counts × provider pricing.
• Latency p50 — median milliseconds per decision, measured. p95/p99 are on the model page.
• ECE — expected calibration error on score cases. Lower means the model’s probabilities mean what they say. ”—” means insufficient score cases.
• Abstain % — how often the model declines. High abstention with high robustness may mean the model is robust because it punts — check benign accuracy on the model page.

How to pick

1. Set your constraints first. Filter by license (open/closed), self-hostable, and price tier. Set a minimum robustness.
2. Sort by ★ robustness / $. The top of that sort is your shortlist.
3. Check the heatmap. If your threat model is prompt injection and the value pick is red there, keep looking.
4. Read the honesty notes. Contamination flags and seller-run numbers live on the model page.

What the number doesn’t mean

Robustness here means resistance to Peira’s 11 attack families on decision tasks. It does not measure general intelligence, helpfulness, or real-world deployment safety. A 90% robust model can still fail on attacks outside these families.