Adapters
An adapter connects a model to the benchmark. Three patterns cover nearly everything.
The interface
def decide(ctx: CaseContext) -> ChoiceOutput | ScoreOutput | AbstainOutput: ...CaseContext carries the case text, options, and decision vocabulary — never gold labels. Adapters that need gold labels to function are rejected; the benchmark is blind by construction.
The three patterns
- Encoder-head (Laya, Kev) — a classifier head over a frozen encoder. Map the head’s argmax to
ChoiceOutput, read logits forScoreOutput. - Logit-readout (SemIf) — run the base LM, read the logits of the decision tokens, renormalize. The adapter does the renormalization, not the model.
- Structured-output API (Jev, LLM baselines) — call the provider with a strict JSON schema, parse the typed decision. Temperature 0, deterministic seeds.
The adapter matrix
| Tier | Adapters |
|---|---|
| Tier 1 (flagship) | Jev, Laya, Kev, SemIf, OpenJev-SGLang |
| Tier 2 (baselines) | GPT-6 Sol/Luna, GPT-5.6 Luna, Grok 4.7, DeepSeek V4.1 Flash, Opus 5.5, Gemini 3.8 Flash, Kimi K3, Claude Fable 5.1 |
| Tier 3 (guardrails) | Shieldstral 1.0, Granite Guardian 4.1, WildGuard, Qwen3-Guard, ShieldGemma 2, gpt-oss-safeguard |
Writing an adapter
pip install peira- Implement
decide(ctx)— see the mock adapter for a worked example - Test offline:
peira run --adapter myadapter --suite trial-demo --verify - Full run:
peira run --adapter myadapter --suite v1-full --seed 20260923 - Seal:
peira seal runs/<run-id>.jsonl— the sealed artifact is your submission
Rules: no network calls inside decide except to your own model endpoint; honor timeouts; deterministic seeds; honest abstention (don’t abstain to dodge hard cases — abstention-flip is measured).