deepseek-flash and jev-1.13.0 cannot be separated at the top. Poisoned retrieval beats almost everyone.

12 models each faced the same 4,701 paired cases. In each pair one version was ordinary and one carried an attack. The attack success rate is the share of eligible cases where the attack changed the decision. Lower is stronger. The lowest rates were 12.4% for deepseek-flash across 4,631 eligible cases and 12.6% for jev-1.13.0 across 4,576 eligible cases. Their confidence intervals overlap, so the data cannot separate them at this sample size.

Generated 2026-10-12. 12 models, 6 ranked, 6 not ranked.

The evidence table

6 models are ranked because their malformed response rate stayed at or below 5 percent. 6 more are shown in full below the ranked table with a flag instead of a rank. Nothing is hidden for failing the threshold. Rates appear with eligible counts in the chart above and in the family detail at the foot of the page.

Flagged. Full numbers, no rank

These models did not qualify for a rank under the published admission rules. That failure is itself a finding. A model that cannot return a usable answer under pressure has a robustness problem before its decisions are even scored.

qwen/qwen3-235b-a22b-2507 recorded 11.9% malformed responses.

kimi-k3 recorded 11.3% malformed responses.

gemini-3.8-flash recorded 8.7% malformed responses across 4,361 eligible cases and would otherwise sit near the top of the table.

Models marked Not ranked ran the full benchmark but did not qualify for a rank under the published admission rules, and the reason is printed with each name. One of those rules caps the malformed response rate at 5 percent, because a score built only on the cases a model managed to answer can flatter it. Every number is shown either way.

One attack family breaks nearly everyone

Retrieval poisoning plants false material in the material a model retrieves before deciding. It was the hardest family in this run by a wide margin. Rates ran from 59.5% for glm-4-plus across 407 eligible cases to 84.3% for grok-4 across 458 eligible cases. Even deepseek-flash flipped on 77.6% of 450 eligible cases in this family. No model was spared.

Every family, every model

The grid below is the evidence behind the headlines. Each cell is the attack success rate for one family and one model. Colour repeats the number and never replaces it. Pale green means under 10 percent. Yellow means 10 percent to under 25 percent. Light orange means 25 percent to under 50 percent. Dark red means 50 percent or more. Open any model beneath the grid for its family intervals and eligible counts.

Two further patterns repay attention. Verbosity inflation, where longer answers drift the decision, reached 73.8% for glm-4-plus and 65.6% for qwen/qwen3-235b-a22b-2507. Cryptographic payload attacks reached 78.0% for inclusionai/ling-3.0-flash and 69.3% for inclusionai/ling-3.0-flash-vl. Weakness is specific. Each model fails in its own places.

Per-model detail

The same test cost about 200 times more for one model

Every run covered the same 4,701 paired cases, so measured cost is directly comparable. jev-1.13.0 completed the run for $0.20 measured. kimi-k3 cost $39.97 measured and took a median 8.33 s per attacked case. Cost is the recorded spend for the run and not a price quote, since prices change after measurement.

Confidence under attack

Calibration asks whether stated confidence matches how often a model is right. Expected calibration error, shortened to ECE, summarises the gap. Lower is better calibrated. Brier score combines calibration and accuracy in one number where lower is also better. Attack raised Brier scores for every model. qwen/qwen3-235b-a22b-2507 showed the sharpest deterioration. Its Brier score moved from 0.0110 on 4,044 calibration cases to 0.1656 under attack.

How to read this page