Microsoft's ThinkingBox Benchmarks AI Agents by the State They Leave Behind

Microsoft has released ThinkingBox, an AI agent evaluation benchmark now available through Hugging Face. Rather than grading the text agents generate, ThinkingBox runs each agent in isolated MCP tool sessions and evaluates the terminal backend state and side effects it leaves behind. It also tests whether the agent can perform consistently 20 times in a row.

The benchmark covers 507 stateful business workflows across retail, auto insurance, travel, neobanking and consulting. Each task starts from an identical clean backend and is run 20 times. In an ablation across 12 LLM models and 121,680 valid trials, 79,853 attempts failed executable checks. Among those failures, 67.24% still terminated cleanly, invoked a state-changing tool and reported no final tool error, yet 77.61% contained wrong field values. Tool usage issues accounted for about 79.9% of failures, rather than reasoning failures.

The results also highlight the gap between single-attempt capability and reliability. Claude Opus 5.5 and Claude Opus 5 each passed 241 of 507 tasks (47.53%) on all 20 attempts. Some open-weight models solved a wider range of tasks at least once but had 20/20 rates below 15%. Microsoft says the grading approach can be applied in production by checking terminal state before committing, rather than relying on a model's summary.

ThinkingBox code is MIT-licensed, the benchmark data is CDLA-Permissive-2.0, and the OpenEnv environment is BSD-3-Clause licensed.