Anthropic's Agent4Sci benchmarks frontier agents on scientific machine learning

Anthropic researchers have introduced Agent4Sci, a benchmarking framework designed to measure how well AI agents can autonomously improve models in scientific machine learning tasks. The framework covers 11 scientific domains, 20 benchmarks and 25 tasks, and the work has been accepted into the Evaluation & Datasets track at NeurIPS 2026.

In experiments, the team evaluated four frontier agents: Claude Opus 4.6, Claude Sonnet 4.6, Gemini 3.1 Pro and Codex (GPT-5.4). Together, the agents improved all 20 benchmarks, with Claude Opus achieving an average relative improvement of 10.4%, rising to 20.1% when three plugin skills were added.

Agent4Sci was built in two stages: benchmarks were selected through a four-phase screening and human standardization process, and a three-phase protocol separates the improvement, evaluation and audit stages to reduce the risk of agents exploiting evaluation loopholes.

Trajectory analysis showed that most gains came from common machine learning engineering techniques such as learning rate adjustments and backbone swaps, rather than from scientific insights. The team then designed three lightweight plugin skills for research ideation, experiment management and periodic checklists, which nearly doubled average improvements for two agents. The researchers say Agent4Sci turns model improvement in automated science into a reproducible and auditable evaluation problem, and helps distinguish gains from engineering skill versus understanding of the underlying science.