Rubric & failure-mode analysis
Survival rate alone does not reveal how an agent thinks. Every session is also scored by a panel of 3 independent judge models against a 16-point rubric, then classified into one of 4 failure modes. This page is the data behind every "agents don't reason causally" claim on the site.
The 16-point rubric
Six criteria across four dimensions, weighted by importance (CR1 and CR2 dominate with weights 5 and 4). Weights sum to 16; the overall score is the weighted mean of the six normalized criterion scores.
| ID | Dimension | Weight | What it measures |
|---|---|---|---|
CR1 | CausalReasoning | 5 | Identify core causal mechanisms in the task report |
CR2 | CausalReasoning | 4 | Identify and avoid traps or spurious correlations |
CR3 | CausalReasoning | 2 | Mechanistic depth and testability |
ED1 | ExperimentalDesign | 2 | Support conclusions with concrete experimental data |
RQ1 | ReflectionQuality | 2 | Acknowledge errors and uncertainties |
DU1 | DataUsage | 1 | Clear data-to-conclusion linkage |
Three judge models (gemini-3-f, grok-4-1-f, qwen3-next) score each session; inter-rater agreement
is high (mean ICC(2,3) = 0.75). Per-criterion scores are averaged across judges and normalized
to [0, 1] by weighted_score / weight.
Per-model rubric profiles
Each model's normalized rubric profile across the six criteria, sorted by overall compliance. Select which models to overlay. The causal axes (CR1–CR3) collapse toward the center on every model — survival rate and causal reasoning are decoupled.
CR1 (5) Identify core causal mechanisms in the task reportCR2 (4) Identify and avoid traps or spurious correlationsCR3 (2) Mechanistic depth and testabilityED1 (2) Support conclusions with concrete experimental dataRQ1 (2) Acknowledge errors and uncertaintiesDU1 (1) Clear data-to-conclusion linkageScores normalized to 0–1 (weighted_score / weight), averaged across the 3 judges. Larger area = stronger causal-reasoning competence. The causal axis (CR1–CR3) is consistently the weakest dimension across all models.
Failure modes
Each judged session is classified into one of four failure patterns by a deterministic rule over its rubric scores (priority A → B → C → D):
| Code | Pattern | Definition | Classification rule |
|---|---|---|---|
| A | No Engagement | No valid reasoning or experimentation | NOT(good_exp_design OR causal_reasoning) |
| B | Blind Exploration | Experiments conducted but no causal mechanism found | good_exp_design AND NOT find_causal_mechanism |
| C | Surface Analysis | Good data utilization but no causal mechanism found | DU1>=1.0 AND NOT find_causal_mechanism |
| D | Nascent Reasoning | Present across all dimensions but weak | (CR1+CR2+CR3) x (ED1 x DU1 x RQ1) > 0 |
Distribution (agentic vs prompting)
ANo Engagement68.4%BBlind Exploration4.2%CSurface Analysis3.3%DNascent Reasoning24.1%
Agentic and prompting sessions fail in the same way: a clear majority never engages causal reasoning, and even the share that reaches nascent reasoning stays weak — causal-reasoning scores are near-zero in every pattern.
Per-model breakdown
| Model | N | A | B | C | D |
|---|---|---|---|---|---|
| Claude-Opus-4.7 | 37 | 1643% | 719% | 00% | 1438% |
| GLM-5.2 | 59 | 2847% | 00% | 35% | 2847% |
| Qwen-3.7-Max | 33 | 1752% | 26% | 412% | 1030% |
| GLM-5.1 | 34 | 1956% | 39% | 13% | 1132% |
| Kimi-K2.6 | 34 | 1750% | 39% | 13% | 1338% |
| Claude-Sonnet-4.5 | 36 | 2261% | 38% | 514% | 617% |
| Hy3-Preview | 39 | 2154% | 410% | 25% | 1231% |
| MiMo-V2.5-Pro | 34 | 1235% | 26% | 515% | 1544% |
| Grok-4.1-Fast | 34 | 2059% | 13% | 26% | 1132% |
| Claude-Opus-4.5 | 31 | 1755% | 13% | 00% | 1342% |
| GPT-5.2-High | 40 | 3178% | 410% | 00% | 513% |
| GPT-5.2 | 41 | 2868% | 25% | 12% | 1024% |
| Gemini-3.5-Flash | 33 | 2061% | 26% | 515% | 618% |
| DeepSeek-V4-Pro | 35 | 2263% | 26% | 26% | 926% |
| DeepSeek-V3.2 | 40 | 3485% | 13% | 00% | 513% |
| GPT-5-Mini | 35 | 2160% | 13% | 00% | 1337% |
| MiniMax-M2 | 33 | 2370% | 39% | 00% | 721% |
| Kimi-K2.5 | 38 | 3182% | 00% | 13% | 616% |
| DeepSeek-V3.2-T | 38 | 3079% | 13% | 00% | 718% |
| MiMo-V2-Flash | 36 | 3597% | 00% | 00% | 13% |
| DeepSeek-V4-Flash | 35 | 2160% | 13% | 26% | 1131% |
| GLM-4.7 | 40 | 2973% | 13% | 00% | 1025% |
| MiniMax-M2-1 | 31 | 2477% | 00% | 00% | 723% |
| MiniMax-M2-7 | 34 | 2574% | 00% | 13% | 824% |
| Grok-4.20 | 40 | 3383% | 13% | 00% | 615% |
| GPT-5.5-XHigh | 33 | 2782% | 13% | 00% | 515% |
| GPT-OSS-120B | 39 | 3385% | 00% | 00% | 615% |
| GPT-5.5-High | 37 | 3286% | 00% | 00% | 514% |
| GPT-5.5 | 37 | 3286% | 00% | 13% | 411% |
| Gemini-3.1-FL | 37 | 3595% | 00% | 00% | 25% |
Per-model counts on agentic sessions. Cell color intensity = share of that model's sessions in each pattern. A (red) dominates almost every row — causal disengagement is uniform across capability tiers.
Dimension scores by failure mode
Within each dimension, bars show the four failure modes. Causal Reasoning stays low for every mode (≤16.3%), whereas Experimental Design, Reflection and Data Usage climb far higher for the engaging modes — the bottleneck is causal reasoning specifically.
Case studies
One representative session per failure mode. Each panel shows the model, scenario, survival outcome and rubric points side by side — then the full per-criterion breakdown with the judge's reasoning and the evidence quoted from the agent's own report. Click any criterion to expand.
ANo Engagement
No engagement: the agent never formed a causal hypothesis nor ran informative experiments. It submits the default high-antenna design with no recorded reasoning beyond the tool call.
Why this matters: this session failed on both survival and causal reasoning (0/16) — the agent never engaged with the problem at all.
Verdict and reasoning shown for one representative judge; criteria scores are averaged across all 3 judges in the aggregate stats. Pattern A classified by: NOT(good_exp_design OR causal_reasoning).
Note how a session can win on survival yet score near zero on causal reasoning — that is precisely the correlation-vs-causation gap CausalGame isolates.