Peira leaderboard

PEIRA
Decision models under attack · suite v2 · dataset 2.6.0
17
MODELS TESTED · 13 RANKED · 4 FLAGGED
4,701
PAIRED CASES EACH · ELIGIBLE 4,297–4,638
10.7–12.4%
LEAD BAND · 7 MODELS TIED AT THE TOP
59.5–84.3%
RETRIEVAL POISONING · LOW END IS FLAGGED GLM-4-PLUS AT 59.5% · NO RANK
≥$0.20 to ≥$59.30
COST SPREAD PER RUN · JEV-1.13.0 (LOWER BOUND) TO GPT-6-ASTRA (LOWER BOUND)

Attack success rates are shares of eligible attacked cases · headline bases 4,297 to 4,638 per model · family bases 354 to 470

Model ranking

Attack success rate with 95% confidence whiskers · lower is stronger · hatched bars carry no rank · a rank requires a malformed response rate at or under 5% · ranks count the 13 ranked models only · the top 7 intervals overlap, a tie · ranks 9 to 11 overlap, a tie · ranks 12 to 13 overlap, a tie · ranks follow point estimates where intervals overlap

Model ranking by attack success rateBars in attack success order. The top 7 intervals overlap, a tie. Ranks 9 to 11 overlap, a tie. Ranks 12 to 13 overlap, a tie. gpt-5.6-terra 10.7% 95% CI 9.8% to 11.6% rank 1, tied band 1 to 7, malformed 0.7%. gpt-6.1-sol 10.7% 95% CI 9.8% to 11.6% rank 2, tied band 1 to 7, malformed 0.1%. kimi-k3 11.0% 95% CI 10.1% to 11.9% rank 3, tied band 1 to 7, malformed 0.0%. gpt-6-astra 11.3% 95% CI 10.4% to 12.2% rank 4, tied band 1 to 7, malformed 0.0%. gemini-3.8-flash 12.2% 95% CI 11.3% to 13.2% rank 5, tied band 1 to 7, malformed 0.3%. gpt-6-luna 12.3% 95% CI 11.4% to 13.3% rank 6, tied band 1 to 7, malformed 2.4%. deepseek-flash 12.4% 95% CI 11.5% to 13.4% rank 7, tied band 1 to 7, malformed 0.0%. jev-1.13.0 12.6% 95% CI 11.7% to 13.6% rank 8, malformed 1.3%. grok-4 16.1% 95% CI 15.0% to 17.2% rank 9, tied band 9 to 11, malformed 0.0%. gpt-5.6-luna 16.5% 95% CI 15.5% to 17.6% rank 10, tied band 9 to 11, malformed 2.2%. solar-pro4 17.4% 95% CI 16.3% to 18.5% rank 11, tied band 9 to 11, malformed 0.6%. ling-3.0-flash-vl 20.9% 95% CI 19.7% to 22.1% flagged no rank, malformed 10.7%. glm-4-plus 21.0% 95% CI 19.8% to 22.2% flagged no rank, malformed 6.2%. claude-sonnet-5-5 21.6% 95% CI 20.4% to 22.8% rank 12, tied band 12 to 13, malformed 1.3%. MiniMax-M3 21.8% 95% CI 20.6% to 23.0% rank 13, tied band 12 to 13, malformed 4.2%. ling-3.0-flash 22.8% 95% CI 21.7% to 24.1% flagged no rank, malformed 7.3%. qwen3-235b-a22b-2507 25.2% 95% CI 23.9% to 26.5% flagged no rank, malformed 11.9%.lower is stronger · hatched bars carry no rank0%5%10%15%20%25%30%1gpt-5.6-terra10.7%malformed 0.7% · tied 1–72gpt-6.1-sol10.7%malformed 0.1% · tied 1–73kimi-k311.0%malformed 0.0% · tied 1–74gpt-6-astra11.3%malformed 0.0% · tied 1–75gemini-3.8-flash12.2%malformed 0.3% · tied 1–76gpt-6-luna12.3%malformed 2.4% · tied 1–77deepseek-flash12.4%malformed 0.0% · tied 1–78jev-1.13.012.6%malformed 1.3%9grok-416.1%malformed 0.0% · tied 9–1110gpt-5.6-luna16.5%malformed 2.2% · tied 9–1111solar-pro417.4%malformed 0.6% · tied 9–11•ling-3.0-flash-vl20.9%no rank · malformed 10.7%•glm-4-plus21.0%no rank · malformed 6.2%12claude-sonnet-5-521.6%malformed 1.3% · tied 12–1313MiniMax-M321.8%malformed 4.2% · tied 12–13•ling-3.0-flash22.8%no rank · malformed 7.3%•qwen3-235b-a22b-250725.2%no rank · malformed 11.9%
Bars run 0% to 30% · lower is stronger
gpt-5.6-terra#1 · TIED 1–710.7%
95% CI 9.8% to 11.6% · malformed 0.7%
gpt-6.1-sol#2 · TIED 1–710.7%
95% CI 9.8% to 11.6% · malformed 0.1%
kimi-k3#3 · TIED 1–711.0%
95% CI 10.1% to 11.9% · malformed 0.0%
gpt-6-astra#4 · TIED 1–711.3%
95% CI 10.4% to 12.2% · malformed 0.0%
gemini-3.8-flash#5 · TIED 1–712.2%
95% CI 11.3% to 13.2% · malformed 0.3%
gpt-6-luna#6 · TIED 1–712.3%
95% CI 11.4% to 13.3% · malformed 2.4%
deepseek-flash#7 · TIED 1–712.4%
95% CI 11.5% to 13.4% · malformed 0.0%
jev-1.13.0#812.6%
95% CI 11.7% to 13.6% · malformed 1.3%
grok-4#9 · TIED 9–1116.1%
95% CI 15.0% to 17.2% · malformed 0.0%
gpt-5.6-luna#10 · TIED 9–1116.5%
95% CI 15.5% to 17.6% · malformed 2.2%
solar-pro4#11 · TIED 9–1117.4%
95% CI 16.3% to 18.5% · malformed 0.6%
ling-3.0-flash-vlFLAGGED20.9%
95% CI 19.7% to 22.1% · no rank · malformed 10.7%
glm-4-plusFLAGGED21.0%
95% CI 19.8% to 22.2% · no rank · malformed 6.2%
claude-sonnet-5-5#12 · TIED 12–1321.6%
95% CI 20.4% to 22.8% · malformed 1.3%
MiniMax-M3#13 · TIED 12–1321.8%
95% CI 20.6% to 23.0% · malformed 4.2%
ling-3.0-flashFLAGGED22.8%
95% CI 21.7% to 24.1% · no rank · malformed 7.3%
qwen3-235b-a22b-2507FLAGGED25.2%
95% CI 23.9% to 26.5% · no rank · malformed 11.9%

Attack family heatmap

Each cell rests on 354 to 470 eligible cases.At this width the last families sit to the right · scroll sideways for them

the remaining families sit to the right · scroll sideways

rows in overall ASR order · cells are family attack success rate in percent, rounded · the top 7 intervals overlap, a tie · ranks 9 to 11 overlap, a tie · ranks 12 to 13 overlap, a tie · ranks follow point estimates where intervals overlap · flagged rows carry no rank · families run by average attack success over the models carrying each family, highest first

Family attack success rates by model in percent. Columns in order, canary leakage, crosslingual, crypto payload, evidence positioning, frequency salience, history leakage, judge sycophancy, question pair incoherence, retrieval poisoning, threshold edge hunting, verbosity inflation.
canary leakagecrosslingualcrypto payloadevidence positioningfrequency saliencehistory leakagejudge sycophancyquestion pair incoherenceretrieval poisoningthreshold edge huntingverbosity inflation
gpt-5.6-terracanary leakage 0.0% · n 417crosslingual 0.2% · n 420crypto payload 16.2% · n 420evidence positioning 4.7% · n 386frequency salience 0.7% · n 420history leakage 1.3% · n 382judge sycophancy 1.2% · n 418question pair incoherence 2.3% · n 399retrieval poisoning 78.8% · n 453threshold edge hunting 0.2% · n 418verbosity inflation 4.9% · n 469
gpt-6.1-solcanary leakage 0.0% · n 420crosslingual 12.1% · n 420crypto payload 16.9% · n 420evidence positioning 5.6% · n 359frequency salience 0.7% · n 420history leakage 1.7% · n 406judge sycophancy 0.5% · n 419question pair incoherence 1.3% · n 396retrieval poisoning 72.8% · n 434threshold edge hunting 0.0% · n 420verbosity inflation 3.0% · n 468
kimi-k3canary leakage 0.0% · n 420crosslingual 0.7% · n 420crypto payload 10.5% · n 420evidence positioning 2.3% · n 400frequency salience 2.4% · n 420history leakage 3.4% · n 411judge sycophancy 1.2% · n 414question pair incoherence 2.3% · n 399retrieval poisoning 82.7% · n 444threshold edge hunting 0.5% · n 420verbosity inflation 9.6% · n 470
gpt-6-astracanary leakage 0.0% · n 420crosslingual 16.4% · n 420crypto payload 22.6% · n 420evidence positioning 4.2% · n 354frequency salience 0.5% · n 420history leakage 1.0% · n 407judge sycophancy 0.5% · n 416question pair incoherence 0.5% · n 395retrieval poisoning 73.8% · n 427threshold edge hunting 0.0% · n 420verbosity inflation 2.4% · n 468
gemini-3.8-flashcanary leakage 0.0% · n 420crosslingual 1.7% · n 420crypto payload 18.8% · n 420evidence positioning 0.6% · n 358frequency salience 1.4% · n 420history leakage 2.2% · n 405judge sycophancy 0.7% · n 413question pair incoherence 2.0% · n 397retrieval poisoning 83.9% · n 448threshold edge hunting 0.0% · n 420verbosity inflation 14.7% · n 470
gpt-6-lunacanary leakage 1.2% · n 419crosslingual 5.2% · n 420crypto payload 31.2% · n 420evidence positioning 3.3% · n 361frequency salience 0.5% · n 420history leakage 1.1% · n 367judge sycophancy 0.7% · n 417question pair incoherence 2.5% · n 397retrieval poisoning 80.5% · n 431threshold edge hunting 0.2% · n 420verbosity inflation 4.9% · n 469
deepseek-flashcanary leakage 0.0% · n 420crosslingual 2.9% · n 420crypto payload 12.1% · n 420evidence positioning 3.0% · n 396frequency salience 0.5% · n 420history leakage 0.3% · n 395judge sycophancy 3.6% · n 420question pair incoherence 1.8% · n 400retrieval poisoning 77.6% · n 450threshold edge hunting 0.0% · n 420verbosity inflation 26.8% · n 470
jev-1.13.0canary leakage 0.0% · n 420crosslingual 15.7% · n 370crypto payload 3.4% · n 411evidence positioning 3.5% · n 396frequency salience 1.0% · n 420history leakage 0.0% · n 404judge sycophancy 3.3% · n 418question pair incoherence 0.0% · n 400retrieval poisoning 83.7% · n 448threshold edge hunting 0.5% · n 419verbosity inflation 20.6% · n 470
grok-4canary leakage 1.2% · n 420crosslingual 13.1% · n 420crypto payload 39.5% · n 420evidence positioning 4.6% · n 372frequency salience 0.2% · n 420history leakage 3.2% · n 407judge sycophancy 3.4% · n 417question pair incoherence 1.3% · n 400retrieval poisoning 84.3% · n 458threshold edge hunting 0.2% · n 417verbosity inflation 17.0% · n 470
gpt-5.6-lunacanary leakage 0.2% · n 418crosslingual 21.9% · n 420crypto payload 60.7% · n 420evidence positioning 4.3% · n 391frequency salience 0.2% · n 420history leakage 1.3% · n 376judge sycophancy 1.9% · n 417question pair incoherence 0.8% · n 400retrieval poisoning 76.1% · n 447threshold edge hunting 0.2% · n 417verbosity inflation 7.7% · n 469
solar-pro4canary leakage 0.7% · n 408crosslingual 28.2% · n 419crypto payload 8.4% · n 417evidence positioning 2.3% · n 400frequency salience 2.4% · n 420history leakage 1.3% · n 396judge sycophancy 4.2% · n 408question pair incoherence 4.0% · n 400retrieval poisoning 76.5% · n 439threshold edge hunting 2.2% · n 404verbosity inflation 51.1% · n 470
ling-3.0-flash-vl · no rankcanary leakage 3.8% · n 420crosslingual 7.4% · n 420crypto payload 69.3% · n 414evidence positioning 5.5% · n 365frequency salience 1.7% · n 420history leakage 2.9% · n 383judge sycophancy 4.5% · n 420question pair incoherence 2.3% · n 394retrieval poisoning 82.6% · n 459threshold edge hunting 1.0% · n 408verbosity inflation 36.7% · n 469
glm-4-plus · no rankcanary leakage 1.6% · n 381crosslingual 53.0% · n 400crypto payload 14.5% · n 408evidence positioning 2.2% · n 401frequency salience 1.5% · n 398history leakage 3.4% · n 387judge sycophancy 8.6% · n 370question pair incoherence 0.8% · n 400retrieval poisoning 59.5% · n 407threshold edge hunting 0.5% · n 406verbosity inflation 73.8% · n 469
claude-sonnet-5-5canary leakage 24.0% · n 420crosslingual 20.5% · n 420crypto payload 97.4% · n 420evidence positioning 2.3% · n 398frequency salience 1.7% · n 417history leakage 4.8% · n 416judge sycophancy 1.2% · n 416question pair incoherence 2.3% · n 398retrieval poisoning 74.5% · n 423threshold edge hunting 0.0% · n 420verbosity inflation 7.5% · n 469
MiniMax-M3canary leakage 3.0% · n 394crosslingual 42.7% · n 415crypto payload 34.5% · n 417evidence positioning 6.6% · n 391frequency salience 2.4% · n 419history leakage 2.1% · n 388judge sycophancy 8.9% · n 381question pair incoherence 4.3% · n 396retrieval poisoning 81.0% · n 422threshold edge hunting 5.2% · n 402verbosity inflation 40.4% · n 455
ling-3.0-flash · no rankcanary leakage 1.7% · n 420crosslingual 17.4% · n 420crypto payload 78.0% · n 419evidence positioning 4.3% · n 374frequency salience 1.7% · n 419history leakage 2.3% · n 388judge sycophancy 4.3% · n 419question pair incoherence 1.3% · n 400retrieval poisoning 82.4% · n 459threshold edge hunting 0.7% · n 417verbosity inflation 44.5% · n 470
qwen3-235b-a22b-2507 · no rankcanary leakage 5.7% · n 388crosslingual 37.8% · n 392crypto payload 26.3% · n 392evidence positioning 8.8% · n 377frequency salience 3.8% · n 398history leakage 2.7% · n 367judge sycophancy 15.3% · n 366question pair incoherence 14.7% · n 360retrieval poisoning 74.9% · n 419threshold edge hunting 7.3% · n 381verbosity inflation 65.6% · n 457
Family attack success rates by model in percent. Columns ordered by average attack success rate, highest first. Averages run over the models carrying each family. retrieval poisoning, crypto payload, verbosity inflation, crosslingual, evidence positioning, judge sycophancy, question pair incoherence, canary leakage, history leakage, frequency salience, threshold edge hunting.
retrieval poisoningcrypto payloadverbosity inflationcrosslingualevidence positioningjudge sycophancyquestion pair incoherencecanary leakagehistory leakagefrequency saliencethreshold edge hunting
gpt-5.6-terraretrieval poisoning 78.8% · n 453crypto payload 16.2% · n 420verbosity inflation 4.9% · n 469crosslingual 0.2% · n 420evidence positioning 4.7% · n 386judge sycophancy 1.2% · n 418question pair incoherence 2.3% · n 399canary leakage 0.0% · n 417history leakage 1.3% · n 382frequency salience 0.7% · n 420threshold edge hunting 0.2% · n 418
gpt-6.1-solretrieval poisoning 72.8% · n 434crypto payload 16.9% · n 420verbosity inflation 3.0% · n 468crosslingual 12.1% · n 420evidence positioning 5.6% · n 359judge sycophancy 0.5% · n 419question pair incoherence 1.3% · n 396canary leakage 0.0% · n 420history leakage 1.7% · n 406frequency salience 0.7% · n 420threshold edge hunting 0.0% · n 420
kimi-k3retrieval poisoning 82.7% · n 444crypto payload 10.5% · n 420verbosity inflation 9.6% · n 470crosslingual 0.7% · n 420evidence positioning 2.3% · n 400judge sycophancy 1.2% · n 414question pair incoherence 2.3% · n 399canary leakage 0.0% · n 420history leakage 3.4% · n 411frequency salience 2.4% · n 420threshold edge hunting 0.5% · n 420
gpt-6-astraretrieval poisoning 73.8% · n 427crypto payload 22.6% · n 420verbosity inflation 2.4% · n 468crosslingual 16.4% · n 420evidence positioning 4.2% · n 354judge sycophancy 0.5% · n 416question pair incoherence 0.5% · n 395canary leakage 0.0% · n 420history leakage 1.0% · n 407frequency salience 0.5% · n 420threshold edge hunting 0.0% · n 420
gemini-3.8-flashretrieval poisoning 83.9% · n 448crypto payload 18.8% · n 420verbosity inflation 14.7% · n 470crosslingual 1.7% · n 420evidence positioning 0.6% · n 358judge sycophancy 0.7% · n 413question pair incoherence 2.0% · n 397canary leakage 0.0% · n 420history leakage 2.2% · n 405frequency salience 1.4% · n 420threshold edge hunting 0.0% · n 420
gpt-6-lunaretrieval poisoning 80.5% · n 431crypto payload 31.2% · n 420verbosity inflation 4.9% · n 469crosslingual 5.2% · n 420evidence positioning 3.3% · n 361judge sycophancy 0.7% · n 417question pair incoherence 2.5% · n 397canary leakage 1.2% · n 419history leakage 1.1% · n 367frequency salience 0.5% · n 420threshold edge hunting 0.2% · n 420
deepseek-flashretrieval poisoning 77.6% · n 450crypto payload 12.1% · n 420verbosity inflation 26.8% · n 470crosslingual 2.9% · n 420evidence positioning 3.0% · n 396judge sycophancy 3.6% · n 420question pair incoherence 1.8% · n 400canary leakage 0.0% · n 420history leakage 0.3% · n 395frequency salience 0.5% · n 420threshold edge hunting 0.0% · n 420
jev-1.13.0retrieval poisoning 83.7% · n 448crypto payload 3.4% · n 411verbosity inflation 20.6% · n 470crosslingual 15.7% · n 370evidence positioning 3.5% · n 396judge sycophancy 3.3% · n 418question pair incoherence 0.0% · n 400canary leakage 0.0% · n 420history leakage 0.0% · n 404frequency salience 1.0% · n 420threshold edge hunting 0.5% · n 419
grok-4retrieval poisoning 84.3% · n 458crypto payload 39.5% · n 420verbosity inflation 17.0% · n 470crosslingual 13.1% · n 420evidence positioning 4.6% · n 372judge sycophancy 3.4% · n 417question pair incoherence 1.3% · n 400canary leakage 1.2% · n 420history leakage 3.2% · n 407frequency salience 0.2% · n 420threshold edge hunting 0.2% · n 417
gpt-5.6-lunaretrieval poisoning 76.1% · n 447crypto payload 60.7% · n 420verbosity inflation 7.7% · n 469crosslingual 21.9% · n 420evidence positioning 4.3% · n 391judge sycophancy 1.9% · n 417question pair incoherence 0.8% · n 400canary leakage 0.2% · n 418history leakage 1.3% · n 376frequency salience 0.2% · n 420threshold edge hunting 0.2% · n 417
solar-pro4retrieval poisoning 76.5% · n 439crypto payload 8.4% · n 417verbosity inflation 51.1% · n 470crosslingual 28.2% · n 419evidence positioning 2.3% · n 400judge sycophancy 4.2% · n 408question pair incoherence 4.0% · n 400canary leakage 0.7% · n 408history leakage 1.3% · n 396frequency salience 2.4% · n 420threshold edge hunting 2.2% · n 404
ling-3.0-flash-vl · no rankretrieval poisoning 82.6% · n 459crypto payload 69.3% · n 414verbosity inflation 36.7% · n 469crosslingual 7.4% · n 420evidence positioning 5.5% · n 365judge sycophancy 4.5% · n 420question pair incoherence 2.3% · n 394canary leakage 3.8% · n 420history leakage 2.9% · n 383frequency salience 1.7% · n 420threshold edge hunting 1.0% · n 408
glm-4-plus · no rankretrieval poisoning 59.5% · n 407crypto payload 14.5% · n 408verbosity inflation 73.8% · n 469crosslingual 53.0% · n 400evidence positioning 2.2% · n 401judge sycophancy 8.6% · n 370question pair incoherence 0.8% · n 400canary leakage 1.6% · n 381history leakage 3.4% · n 387frequency salience 1.5% · n 398threshold edge hunting 0.5% · n 406
claude-sonnet-5-5retrieval poisoning 74.5% · n 423crypto payload 97.4% · n 420verbosity inflation 7.5% · n 469crosslingual 20.5% · n 420evidence positioning 2.3% · n 398judge sycophancy 1.2% · n 416question pair incoherence 2.3% · n 398canary leakage 24.0% · n 420history leakage 4.8% · n 416frequency salience 1.7% · n 417threshold edge hunting 0.0% · n 420
MiniMax-M3retrieval poisoning 81.0% · n 422crypto payload 34.5% · n 417verbosity inflation 40.4% · n 455crosslingual 42.7% · n 415evidence positioning 6.6% · n 391judge sycophancy 8.9% · n 381question pair incoherence 4.3% · n 396canary leakage 3.0% · n 394history leakage 2.1% · n 388frequency salience 2.4% · n 419threshold edge hunting 5.2% · n 402
ling-3.0-flash · no rankretrieval poisoning 82.4% · n 459crypto payload 78.0% · n 419verbosity inflation 44.5% · n 470crosslingual 17.4% · n 420evidence positioning 4.3% · n 374judge sycophancy 4.3% · n 419question pair incoherence 1.3% · n 400canary leakage 1.7% · n 420history leakage 2.3% · n 388frequency salience 1.7% · n 419threshold edge hunting 0.7% · n 417
qwen3-235b-a22b-2507 · no rankretrieval poisoning 74.9% · n 419crypto payload 26.3% · n 392verbosity inflation 65.6% · n 457crosslingual 37.8% · n 392evidence positioning 8.8% · n 377judge sycophancy 15.3% · n 366question pair incoherence 14.7% · n 360canary leakage 5.7% · n 388history leakage 2.7% · n 367frequency salience 3.8% · n 398threshold edge hunting 7.3% · n 381

Cost vs attack success rate

Log cost axis, each step right multiplies cost by ten · one point per full 4,701 case run · filled points are ranked, hollow points are flagged · a pale dot in the centre marks a lower bound cost · cheap and strong sits bottom left · the top 7 intervals overlap, a tie · ranks 9 to 11 overlap, a tie · ranks 12 to 13 overlap, a tie · ranks follow point estimates where intervals overlap

Cost per run against attack success rateCost per run against attack success rate. The cost axis is log scale. Filled points are ranked models and hollow points are flagged models with no rank. Points at one cost and rate, or too near to draw apart, are offset by a marker width at most so each stays visible. gpt-5.6-terra $12.83 (lower bound) at 10.7%, rank 1, tied band 1 to 7. gpt-6.1-sol $12.27 (lower bound) at 10.7%, rank 2, tied band 1 to 7. kimi-k3 $21.02 at 11.0%, rank 3, tied band 1 to 7. gpt-6-astra $59.30 (lower bound) at 11.3%, rank 4, tied band 1 to 7. gemini-3.8-flash $3.17 (lower bound) at 12.2%, rank 5, tied band 1 to 7. gpt-6-luna $0.75 (lower bound) at 12.3%, rank 6, tied band 1 to 7. deepseek-flash $1.48 at 12.4%, rank 7, tied band 1 to 7. jev-1.13.0 $0.20 (lower bound) at 12.6%, rank 8. grok-4 $11.09 at 16.1%, rank 9, tied band 9 to 11. gpt-5.6-luna $1.51 (lower bound) at 16.5%, rank 10, tied band 9 to 11. solar-pro4 $0.45 (lower bound) at 17.4%, rank 11, tied band 9 to 11. ling-3.0-flash-vl $0.26 (lower bound) at 20.9%, flagged no rank. glm-4-plus $4.27 (lower bound) at 21.0%, flagged no rank. claude-sonnet-5-5 $26.79 (lower bound) at 21.6%, rank 12, tied band 12 to 13. MiniMax-M3 $3.62 (lower bound) at 21.8%, rank 13, tied band 12 to 13. ling-3.0-flash $0.24 (lower bound) at 22.8%, flagged no rank. qwen3-235b-a22b-2507 $0.46 (lower bound) at 25.2%, flagged no rank.$0.1$1$10$100cost per run in USD, log scaleattack success rate0%10%20%30%tied bands 1–7 · 9–11 · 12–13dots mark the lower bound costscrowded points are offset by a marker width at mostrankedflagged · no ranklower bound costgpt-5.6-terragpt-6.1-solkimi-k3gpt-6-astragemini-3.8-flashgpt-6-lunadeepseek-flashjev-1.13.0grok-4gpt-5.6-lunasolar-pro4claude-sonnet-5-5MiniMax-M3ling-3.0-flash-vlglm-4-plusling-3.0-flashqwen3-235b-a22b-2507Cost per run against attack success rateCost per run against attack success rate. The cost axis is log scale. Filled points are ranked models and hollow points are flagged models with no rank. Points at one cost and rate, or too near to draw apart, are offset by a marker width at most so each stays visible. gpt-5.6-terra $12.83 (lower bound) at 10.7%, rank 1, tied band 1 to 7. gpt-6.1-sol $12.27 (lower bound) at 10.7%, rank 2, tied band 1 to 7. kimi-k3 $21.02 at 11.0%, rank 3, tied band 1 to 7. gpt-6-astra $59.30 (lower bound) at 11.3%, rank 4, tied band 1 to 7. gemini-3.8-flash $3.17 (lower bound) at 12.2%, rank 5, tied band 1 to 7. gpt-6-luna $0.75 (lower bound) at 12.3%, rank 6, tied band 1 to 7. deepseek-flash $1.48 at 12.4%, rank 7, tied band 1 to 7. jev-1.13.0 $0.20 (lower bound) at 12.6%, rank 8. grok-4 $11.09 at 16.1%, rank 9, tied band 9 to 11. gpt-5.6-luna $1.51 (lower bound) at 16.5%, rank 10, tied band 9 to 11. solar-pro4 $0.45 (lower bound) at 17.4%, rank 11, tied band 9 to 11. ling-3.0-flash-vl $0.26 (lower bound) at 20.9%, flagged no rank. glm-4-plus $4.27 (lower bound) at 21.0%, flagged no rank. claude-sonnet-5-5 $26.79 (lower bound) at 21.6%, rank 12, tied band 12 to 13. MiniMax-M3 $3.62 (lower bound) at 21.8%, rank 13, tied band 12 to 13. ling-3.0-flash $0.24 (lower bound) at 22.8%, flagged no rank. qwen3-235b-a22b-2507 $0.46 (lower bound) at 25.2%, flagged no rank.$0.1$1$10$100cost per run in USD, log scaleattack success rate0%10%20%30%tied bands 1–7 · 9–11 · 12–13dots mark the lower bound costscrowded points are offset by a marker width at mostrankedflagged · no ranklower bound costgpt-5.6-terragpt-6.1-solkimi-k3gpt-6-astragemini-3.8-flashgpt-6-lunadeepseek-flashjev-1.13.0grok-4gpt-5.6-lunasolar-pro4claude-sonnet-5-5MiniMax-M3ling-3.0-flash-vlglm-4-plusling-3.0-flashqwen3-235b-a22b-2507

Malformed response rate

Fraction of the 4,701 cases with any malformed call record, benign or attacked · the admission rule for earning a rank · amber names are flagged models with no rank

Malformed response rate by modelMalformed response rate by model, worst first. qwen3-235b-a22b-2507 11.9%, flagged no rank. ling-3.0-flash-vl 10.7%, flagged no rank. ling-3.0-flash 7.3%, flagged no rank. glm-4-plus 6.2%, flagged no rank. MiniMax-M3 4.2%, rank 13, tied band 12 to 13. gpt-6-luna 2.4%, rank 6, tied band 1 to 7. gpt-5.6-luna 2.2%, rank 10, tied band 9 to 11. jev-1.13.0 1.3%, rank 8. claude-sonnet-5-5 1.3%, rank 12, tied band 12 to 13. gpt-5.6-terra 0.7%, rank 1, tied band 1 to 7. solar-pro4 0.6%, rank 11, tied band 9 to 11. gemini-3.8-flash 0.3%, rank 5, tied band 1 to 7. gpt-6.1-sol 0.1%, rank 2, tied band 1 to 7. gpt-6-astra 0.0%, rank 4, tied band 1 to 7. kimi-k3 0.0%, rank 3, tied band 1 to 7. deepseek-flash 0.0%, rank 7, tied band 1 to 7. grok-4 0.0%, rank 9, tied band 9 to 11. The dashed line marks the 5% admission line. Bars past the line are amber and bars under it are green.worst first• qwen3-235b-a22b-2507• ling-3.0-flash-vl• ling-3.0-flash• glm-4-plusMiniMax-M3gpt-6-lunagpt-5.6-lunajev-1.13.0claude-sonnet-5-5gpt-5.6-terrasolar-pro4gemini-3.8-flashgpt-6.1-solgpt-6-astrakimi-k3deepseek-flashgrok-411.9%10.7%7.3%6.2%4.2%2.4%2.2%1.3%1.3%0.7%0.6%0.3%0.1%0.0%0.0%0.0%0.0%5% admission line0%10%13%bars past the line are over the 5% admission limit
Bars run 0% to 13% · the dashed tick is the 5% admission line · bars past it are over the 5% admission limit
qwen3-235b-a22b-250711.9%, flagged, no rank
ling-3.0-flash-vl10.7%, flagged, no rank
ling-3.0-flash7.3%, flagged, no rank
glm-4-plus6.2%, flagged, no rank
MiniMax-M34.2%, rank 13 of 13 ranked, tied band 12 to 13
gpt-6-luna2.4%, rank 6 of 13 ranked, tied band 1 to 7
gpt-5.6-luna2.2%, rank 10 of 13 ranked, tied band 9 to 11
jev-1.13.01.3%, rank 8 of 13 ranked
claude-sonnet-5-51.3%, rank 12 of 13 ranked, tied band 12 to 13
gpt-5.6-terra0.7%, rank 1 of 13 ranked, tied band 1 to 7
solar-pro40.6%, rank 11 of 13 ranked, tied band 9 to 11
gemini-3.8-flash0.3%, rank 5 of 13 ranked, tied band 1 to 7
gpt-6.1-sol0.1%, rank 2 of 13 ranked, tied band 1 to 7
gpt-6-astra0.0%, rank 4 of 13 ranked, tied band 1 to 7
kimi-k30.0%, rank 3 of 13 ranked, tied band 1 to 7
deepseek-flash0.0%, rank 7 of 13 ranked, tied band 1 to 7
grok-40.0%, rank 9 of 13 ranked, tied band 9 to 11

Model cards

Spark bars show the 11 family ASRs in fixed order (canary leakage, crosslingual, crypto payload, evidence positioning, frequency salience, history leakage, judge sycophancy, question pair incoherence, retrieval poisoning, threshold edge hunting, verbosity inflation) on a shared 0 to 100% scale · bar fill is green on ranked cards and grey on flagged cards, apart from the orange retrieval spike · a dotted outline marks a family a model does not carry · a nonzero rate always draws at least a 1px sliver, so a missing bar is a true zero and small bars are read by their printed values · the orange spike is retrieval poisoning, bar 9 of 11 · latency and calibration are measured on the attacked runs, where lower ECE and Brier scores are better · calibration rests on the cases that reported a confidence · C·H·M is critical, high, medium · robust tax adds the benign accuracy and attacked ECE shortfalls from the best ranked runs, in points · severity flip rate confidence intervals reach about 2.8 points to one side at the widest

gpt-5.6-terra

RANK 1 OF 13 RANKED · TIED 1–7
Malformed response rate at or under 5%
10.7%ASR · 95% CI 9.8% to 11.6%
Orange is retrieval poisoning · no other family reaches 20%
ELIGIBLE CASES4,602
BENIGN ACC · DECIDED VARIANTS97.9%
COST · LOWER BOUND$12.83
LATENCY P50 · ATTACKED1.95s
LATENCY P95 · ATTACKED3.63s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,568 · 99.3%
ECE · ATTACKED0.072
BRIER · ATTACKED0.083
ROBUST TAX · VS BEST RANKED RUNS8.3 pts
MALFORMED0.7%
SEV FLIP RATE C·H·M8.4%·10.0%·12.3%
SEV BASES C·H·M1,032·1,542·2,028

gpt-6.1-sol

RANK 2 OF 13 RANKED · TIED 1–7
Malformed response rate at or under 5%
10.7%ASR · 95% CI 9.8% to 11.6%
Orange is retrieval poisoning · no other family reaches 20%
ELIGIBLE CASES4,582
BENIGN ACC · DECIDED VARIANTS97.5%
COST · LOWER BOUND$12.27
LATENCY P50 · ATTACKED4.25s
LATENCY P95 · ATTACKED9.42s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,578 · 99.9%
ECE · ATTACKED0.081
BRIER · ATTACKED0.092
ROBUST TAX · VS BEST RANKED RUNS8.3 pts
MALFORMED0.1%
SEV FLIP RATE C·H·M7.9%·10.6%·12.1%
SEV BASES C·H·M1,026·1,539·2,017

kimi-k3

RANK 3 OF 13 RANKED · TIED 1–7
Malformed response rate at or under 5%
First 3,274 cases re-run under the fixed adapter · see Notes
11.0%ASR · 95% CI 10.1% to 11.9%
Orange is retrieval poisoning · no other family reaches 20%
ELIGIBLE CASES4,638
BENIGN ACC · DECIDED VARIANTS98.7%
COST$21.02
LATENCY P50 · ATTACKED3.85s
LATENCY P95 · ATTACKED6.00s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,638 · 100.0%
ECE · ATTACKED0.064
BRIER · ATTACKED0.086
ROBUST TAX · VS BEST RANKED RUNS1.6 pts
MALFORMED0.0%
SEV FLIP RATE C·H·M7.7%·9.8%·13.4%
SEV BASES C·H·M1,035·1,550·2,053

gpt-6-astra

RANK 4 OF 13 RANKED · TIED 1–7
Malformed response rate at or under 5%
11.3%ASR · 95% CI 10.4% to 12.2%
Orange is retrieval poisoning · top other spike is crypto payload 22.6%
ELIGIBLE CASES4,567
BENIGN ACC · DECIDED VARIANTS97.2%
COST · LOWER BOUND$59.30
LATENCY P50 · ATTACKED4.47s
LATENCY P95 · ATTACKED8.96s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,566 · 99.98%
ECE · ATTACKED0.087
BRIER · ATTACKED0.099
ROBUST TAX · VS BEST RANKED RUNS8.3 pts
MALFORMED0.0%
SEV FLIP RATE C·H·M8.6%·11.2%·12.6%
SEV BASES C·H·M1,022·1,529·2,016

gemini-3.8-flash

RANK 5 OF 13 RANKED · TIED 1–7
Malformed response rate at or under 5%
First 1,610 cases re-run under the fixed adapter · see Notes
12.2%ASR · 95% CI 11.3% to 13.2%
Orange is retrieval poisoning · no other family reaches 20%
ELIGIBLE CASES4,591
BENIGN ACC · DECIDED VARIANTS97.7%
COST · LOWER BOUND$3.17
LATENCY P50 · ATTACKED2.37s
LATENCY P95 · ATTACKED6.85s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,527 · 98.6%
ECE · ATTACKED0.080
BRIER · ATTACKED0.093
ROBUST TAX · VS BEST RANKED RUNS4.2 pts
MALFORMED0.3%
SEV FLIP RATE C·H·M9.7%·10.4%·14.8%
SEV BASES C·H·M1,029·1,544·2,018

gpt-6-luna

RANK 6 OF 13 RANKED · TIED 1–7
Malformed response rate at or under 5%
12.3%ASR · 95% CI 11.4% to 13.3%
Orange is retrieval poisoning · top other spike is crypto payload 31.2%
ELIGIBLE CASES4,541
BENIGN ACC · DECIDED VARIANTS96.6%
COST · LOWER BOUND$0.75
LATENCY P50 · ATTACKED2.20s
LATENCY P95 · ATTACKED5.30s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,433 · 97.6%
ECE · ATTACKED0.077
BRIER · ATTACKED0.087
ROBUST TAX · VS BEST RANKED RUNS8.3 pts
MALFORMED2.4%
SEV FLIP RATE C·H·M12.8%·12.0%·12.4%
SEV BASES C·H·M1,027·1,540·1,974

deepseek-flash

RANK 7 OF 13 RANKED · TIED 1–7
Malformed response rate at or under 5%
12.4%ASR · 95% CI 11.5% to 13.4%
Orange is retrieval poisoning · top other spike is verbosity inflation 26.8%
ELIGIBLE CASES4,631
BENIGN ACC · DECIDED VARIANTS98.5%
COST$1.48
LATENCY P50 · ATTACKED1.31s
LATENCY P95 · ATTACKED1.85s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,631 · 100.0%
ECE · ATTACKED0.071
BRIER · ATTACKED0.094
ROBUST TAX · VS BEST RANKED RUNS2.4 pts
MALFORMED0.0%
SEV FLIP RATE C·H·M7.8%·11.0%·15.8%
SEV BASES C·H·M1,037·1,549·2,045

jev-1.13.0

RANK 8 OF 13 RANKED
Malformed response rate at or under 5%
12.6%ASR · 95% CI 11.7% to 13.6%
Orange is retrieval poisoning · top other spike is verbosity inflation 20.6%
ELIGIBLE CASES4,576
BENIGN ACC · DECIDED VARIANTS98.6%
COST · LOWER BOUND$0.20
LATENCY P50 · ATTACKED594ms
LATENCY P95 · ATTACKED1.14s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,576 · 100.0%
ECE · ATTACKED0.096
BRIER · ATTACKED0.118
ROBUST TAX · VS BEST RANKED RUNS4.9 pts
MALFORMED1.3%
SEV FLIP RATE C·H·M8.8%·10.4%·16.3%
SEV BASES C·H·M1,035·1,539·2,002

grok-4

RANK 9 OF 13 RANKED · TIED 9–11
Malformed response rate at or under 5%
16.1%ASR · 95% CI 15.0% to 17.2%
Orange is retrieval poisoning · top other spike is crypto payload 39.5%
ELIGIBLE CASES4,621
BENIGN ACC · DECIDED VARIANTS98.3%
COST$11.09
LATENCY P50 · ATTACKED4.45s
LATENCY P95 · ATTACKED8.85s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,621 · 100.0%
ECE · ATTACKED0.108
BRIER · ATTACKED0.126
ROBUST TAX · VS BEST RANKED RUNS6.3 pts
MALFORMED0.0%
SEV FLIP RATE C·H·M14.4%·14.5%·18.1%
SEV BASES C·H·M1,036·1,554·2,031

gpt-5.6-luna

RANK 10 OF 13 RANKED · TIED 9–11
Malformed response rate at or under 5%
16.5%ASR · 95% CI 15.5% to 17.6%
Orange is retrieval poisoning · top other spike is crypto payload 60.7%
ELIGIBLE CASES4,595
BENIGN ACC · DECIDED VARIANTS97.8%
COST · LOWER BOUND$1.51
LATENCY P50 · ATTACKED2.13s
LATENCY P95 · ATTACKED4.68s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,495 · 97.8%
ECE · ATTACKED0.122
BRIER · ATTACKED0.130
ROBUST TAX · VS BEST RANKED RUNS8.3 pts
MALFORMED2.2%
SEV FLIP RATE C·H·M17.9%·16.3%·16.0%
SEV BASES C·H·M1,036·1,544·2,015

solar-pro4

RANK 11 OF 13 RANKED · TIED 9–11
Malformed response rate at or under 5%
Model ID · upstage/solar-pro4
17.4%ASR · 95% CI 16.3% to 18.5%
Orange is retrieval poisoning · top other spike is verbosity inflation 51.1%
ELIGIBLE CASES4,581
BENIGN ACC · DECIDED VARIANTS97.7%
COST · LOWER BOUND$0.45
LATENCY P50 · ATTACKED2.33s
LATENCY P95 · ATTACKED5.69s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,564 · 99.6%
ECE · ATTACKED0.125
BRIER · ATTACKED0.145
ROBUST TAX · VS BEST RANKED RUNS8.6 pts
MALFORMED0.6%
SEV FLIP RATE C·H·M14.5%·14.3%·21.3%
SEV BASES C·H·M1,029·1,541·2,011

ling-3.0-flash-vl

FLAGGED · NO RANK
Malformed response rate above 5%
Model ID · inclusionai/ling-3.0-flash-vl
20.9%ASR · 95% CI 19.7% to 22.1%
Orange is retrieval poisoning · top other spike is crypto payload 69.3%
ELIGIBLE CASES4,572
BENIGN ACC · DECIDED VARIANTS98.3%
COST · LOWER BOUND$0.26
LATENCY P50 · ATTACKED2.07s
LATENCY P95 · ATTACKED5.30s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,131 · 90.4%
ECE · ATTACKED0.068
BRIER · ATTACKED0.090
ROBUST TAX · VS BEST RANKED RUNSn/a
MALFORMED10.7%
SEV FLIP RATE C·H·M21.5%·18.6%·22.3%
SEV BASES C·H·M1,022·1,528·2,022

glm-4-plus

FLAGGED · NO RANK
Malformed response rate above 5%
21.0%ASR · 95% CI 19.8% to 22.2%
Orange is retrieval poisoning · top spike is verbosity inflation 73.8%
ELIGIBLE CASES4,427
BENIGN ACC · DECIDED VARIANTS98.9%
COST · LOWER BOUND$4.27
LATENCY P50 · ATTACKED2.03s
LATENCY P95 · ATTACKED4.05s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,361 · 98.5%
ECE · ATTACKED0.124
BRIER · ATTACKED0.148
ROBUST TAX · VS BEST RANKED RUNSn/a
MALFORMED6.2%
SEV FLIP RATE C·H·M19.6%·17.0%·24.9%
SEV BASES C·H·M1,019·1,511·1,897

claude-sonnet-5-5

RANK 12 OF 13 RANKED · TIED 12–13
Malformed response rate at or under 5%
21.6%ASR · 95% CI 20.4% to 22.8%
Orange is retrieval poisoning · top spike is crypto payload 97.4%
ELIGIBLE CASES4,617
BENIGN ACC · DECIDED VARIANTS98.3%
COST · LOWER BOUND$26.79
LATENCY P50 · ATTACKED2.88s
LATENCY P95 · ATTACKED8.14s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,084 · 88.5%
ECE · ATTACKED0.048
BRIER · ATTACKED0.076
ROBUST TAX · VS BEST RANKED RUNS0.4 pts
MALFORMED1.3%
SEV FLIP RATE C·H·M27.0%·21.4%·19.0%
SEV BASES C·H·M1,030·1,545·2,042

MiniMax-M3

RANK 13 OF 13 RANKED · TIED 12–13
Malformed response rate at or under 5%
21.8%ASR · 95% CI 20.6% to 23.0%
Orange is retrieval poisoning · top other spike is crosslingual 42.7%
ELIGIBLE CASES4,480
BENIGN ACC · DECIDED VARIANTS97.8%
COST · LOWER BOUND$3.62
LATENCY P50 · ATTACKED2.84s
LATENCY P95 · ATTACKED6.58s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,408 · 98.4%
ECE · ATTACKED0.107
BRIER · ATTACKED0.152
ROBUST TAX · VS BEST RANKED RUNS6.7 pts
MALFORMED4.2%
SEV FLIP RATE C·H·M19.9%·17.8%·25.8%
SEV BASES C·H·M1,033·1,498·1,949

ling-3.0-flash

FLAGGED · NO RANK
Malformed response rate above 5%
Model ID · inclusionai/ling-3.0-flash
22.8%ASR · 95% CI 21.7% to 24.1%
Orange is retrieval poisoning · top other spike is crypto payload 78.0%
ELIGIBLE CASES4,605
BENIGN ACC · DECIDED VARIANTS98.3%
COST · LOWER BOUND$0.24
LATENCY P50 · ATTACKED1.96s
LATENCY P95 · ATTACKED6.23s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,281 · 93.0%
ECE · ATTACKED0.123
BRIER · ATTACKED0.138
ROBUST TAX · VS BEST RANKED RUNSn/a
MALFORMED7.3%
SEV FLIP RATE C·H·M23.8%·21.2%·23.7%
SEV BASES C·H·M1,034·1,550·2,021

qwen3-235b-a22b-2507

FLAGGED · NO RANK
Malformed response rate above 5%
Model ID · qwen/qwen3-235b-a22b-2507
25.2%ASR · 95% CI 23.9% to 26.5%
Orange is retrieval poisoning · top other spike is verbosity inflation 65.6%
ELIGIBLE CASES4,297
BENIGN ACC · DECIDED VARIANTS97.8%
COST · LOWER BOUND$0.46
LATENCY P50 · ATTACKED4.88s
LATENCY P95 · ATTACKED20.01s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,044 · 94.1%
ECE · ATTACKED0.144
BRIER · ATTACKED0.166
ROBUST TAX · VS BEST RANKED RUNSn/a
MALFORMED11.9%
SEV FLIP RATE C·H·M21.0%·21.4%·30.2%
SEV BASES C·H·M965·1,445·1,887