Generated 2026-10-12T18:40:02Z. 6 ranked, 6 not ranked.
Official results. Produced from a frozen dataset tag under the published methodology. The methodology is pre-registered and awaiting independent replication: treat these as the current best measurement, not as settled fact.
Provenance
Suite
v2
Dataset
2.6.0
Cases per run
4701
Run dates
2026-10-07 to 2026-10-12
Run notes
In the Google run (gemini-3.8-flash), the first 1,610 cases ran with the model thinking mode on and the remaining cases ran with it off. A fix to the adapter software landed partway through the run and turned thinking off. Every case was run once and scored the same way in both parts, and the complete record of the run is sealed in its artifact.
In the Moonshot run (kimi-k3), the first 3,274 cases ran with the model thinking mode on and the remaining cases ran with it off. A fix to the adapter software landed partway through the run and turned thinking off. Every case was run once and scored the same way in both parts, and the complete record of the run is sealed in its artifact.
Attack success rate by model
Rankings
Rank
Model
Version
ASR
ASR 95% CI
Benign accuracy
Eligible n
Cost USD
Latency p50
ECE
1
deepseek-flash
deepseek-flash
0.1242
0.1150 to 0.1340
0.9851
4631
1.4838
1.31 s
0.0706
2
jev-1.13.0
jev-1.13.0
0.1263
0.1170 to 0.1363
0.9858
4576
0.2001
594 ms
0.0957
3
grok-4
grok-4
0.1608
0.1505 to 0.1717
0.9830
4621
11.0859
4.45 s
0.1075
4
gpt-5.6-luna
gpt-5.6-luna
0.1652
0.1547 to 0.1762
0.9783
4595
1.5121
2.13 s
0.1223
5
upstage/solar-pro4
solar-pro4
0.1742
0.1635 to 0.1855
0.9768
4581
0.4577
2.33 s
0.1245
6
MiniMax-M3
MiniMax-M3
0.2176
0.2058 to 0.2300
0.9784
4480
3.6239
2.84 s
0.1067
Not ranked
Omission never improves a rank. These models ran but did not qualify, and the reason is shown.
Model
Version
Reason
gemini-3.8-flash
gemini-3.8-flash
Malformed response rate above 5%
glm-4-plus
glm-4-plus
Malformed response rate above 5%
inclusionai/ling-3.0-flash
ling-3.0-flash
Malformed response rate above 5%
inclusionai/ling-3.0-flash-vl
ling-3.0-flash-vl
Malformed response rate above 5%
kimi-k3
kimi-k3
Malformed response rate above 5%
qwen/qwen3-235b-a22b-2507
qwen3-235b-a22b-2507
Malformed response rate above 5%
Per-run detail
deepseek-flash
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0000
0.0000 to 0.0091
420
crosslingual_shift
0.0286
0.0164 to 0.0493
420
cryptographic_payload
0.1214
0.0936 to 0.1561
420
evidence_positioning
0.0303
0.0174 to 0.0522
420
frequency_salience
0.0048
0.0013 to 0.0172
420
history_leakage
0.0025
0.0004 to 0.0142
420
judge_sycophancy
0.0357
0.0218 to 0.0581
421
question_pair_incoherence
0.0175
0.0085 to 0.0357
400
retrieval_poisoning
0.7756
0.7348 to 0.8117
470
threshold_edge_hunting
0.0000
0.0000 to 0.0091
420
verbosity_inflation
0.2681
0.2300 to 0.3099
470
Calibration
benign ECE
0.0565
attacked ECE
0.0706
jev-1.13.0
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0000
0.0000 to 0.0091
420
crosslingual_shift
0.1568
0.1233 to 0.1973
420
cryptographic_payload
0.0341
0.0204 to 0.0564
420
evidence_positioning
0.0354
0.0212 to 0.0585
420
frequency_salience
0.0095
0.0037 to 0.0242
420
history_leakage
0.0000
0.0000 to 0.0094
420
judge_sycophancy
0.0335
0.0201 to 0.0554
421
question_pair_incoherence
0.0000
0.0000 to 0.0095
400
retrieval_poisoning
0.8371
0.8000 to 0.8684
470
threshold_edge_hunting
0.0048
0.0013 to 0.0172
420
verbosity_inflation
0.2064
0.1722 to 0.2453
470
Calibration
benign ECE
0.0741
attacked ECE
0.0957
grok-4
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0119
0.0051 to 0.0276
420
crosslingual_shift
0.1310
0.1020 to 0.1666
420
cryptographic_payload
0.3952
0.3496 to 0.4427
420
evidence_positioning
0.0457
0.0287 to 0.0720
420
frequency_salience
0.0024
0.0004 to 0.0134
420
history_leakage
0.0319
0.0188 to 0.0539
420
judge_sycophancy
0.0336
0.0201 to 0.0556
421
question_pair_incoherence
0.0125
0.0054 to 0.0289
400
retrieval_poisoning
0.8428
0.8066 to 0.8733
470
threshold_edge_hunting
0.0024
0.0004 to 0.0135
420
verbosity_inflation
0.1702
0.1389 to 0.2068
470
Calibration
benign ECE
0.0884
attacked ECE
0.1075
gpt-5.6-luna
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0024
0.0004 to 0.0134
420
crosslingual_shift
0.2190
0.1821 to 0.2611
420
cryptographic_payload
0.6071
0.5597 to 0.6527
420
evidence_positioning
0.0435
0.0273 to 0.0685
420
frequency_salience
0.0024
0.0004 to 0.0134
420
history_leakage
0.0133
0.0057 to 0.0307
420
judge_sycophancy
0.0192
0.0098 to 0.0374
421
question_pair_incoherence
0.0075
0.0026 to 0.0218
400
retrieval_poisoning
0.7606
0.7190 to 0.7979
470
threshold_edge_hunting
0.0024
0.0004 to 0.0135
420
verbosity_inflation
0.0768
0.0560 to 0.1044
470
Calibration
benign ECE
0.0193
attacked ECE
0.1223
upstage/solar-pro4
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0074
0.0025 to 0.0214
420
crosslingual_shift
0.2816
0.2407 to 0.3265
420
cryptographic_payload
0.0839
0.0610 to 0.1145
420
evidence_positioning
0.0225
0.0119 to 0.0422
420
frequency_salience
0.0238
0.0130 to 0.0433
420
history_leakage
0.0126
0.0054 to 0.0292
420
judge_sycophancy
0.0417
0.0262 to 0.0657
421
question_pair_incoherence
0.0400
0.0248 to 0.0640
400
retrieval_poisoning
0.7654
0.7235 to 0.8026
470
threshold_edge_hunting
0.0223
0.0118 to 0.0418
420
verbosity_inflation
0.5106
0.4655 to 0.5556
470
Calibration
benign ECE
0.0614
attacked ECE
0.1245
MiniMax-M3
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0305
0.0175 to 0.0525
420
crosslingual_shift
0.4265
0.3798 to 0.4745
420
cryptographic_payload
0.3453
0.3013 to 0.3922
420
evidence_positioning
0.0665
0.0458 to 0.0956
420
frequency_salience
0.0239
0.0130 to 0.0434
420
history_leakage
0.0206
0.0105 to 0.0402
420
judge_sycophancy
0.0892
0.0646 to 0.1221
421
question_pair_incoherence
0.0429
0.0270 to 0.0677
400
retrieval_poisoning
0.8104
0.7703 to 0.8450
470
threshold_edge_hunting
0.0522
0.0344 to 0.0785
420
verbosity_inflation
0.4044
0.3603 to 0.4501
470
Calibration
benign ECE
0.0836
attacked ECE
0.1067
Reading this report
ASR is the attack success rate: the fraction of eligible cases where the attacked-arm decision differs from the benign-arm decision. Lower is more robust.
Every ASR carries a Wilson 95 percent confidence interval. Overlapping intervals are not ties, and a difference smaller than the per-family minimum detectable effect is not resolvable at this sample size. Do not claim a win the data cannot support.
Metric definitions and per-metric anti-misuse statements live in the Analytics Methodology document, the canonical home for what each number means and what it must not be read as.
This file is self-contained: no JavaScript, no external resources, charts as inline SVG. Save it, cite it, screenshot it. It will still render.