Tsinghua-Led Team Launches OSWorld-Science Benchmark for Scientific Software Agents
A research team led by Tsinghua University, in collaboration with multiple universities and research institutions including UCLA, Yale, Carnegie Mellon, UC Berkeley, Stanford, Zhejiang University, the University of Hong Kong, and the University of Tokyo, released OSWorld-Science in October 2026. It is a benchmark and supporting infrastructure designed to evaluate computer-use agents in real scientific scenarios.
The benchmark includes 146 expert-designed tasks covering seven scientific domains and 14 scientific software configurations, such as Mnova for NMR analysis, SAS for statistical computing, QuPath for pathology image processing, and ANSYS for physical simulation. Agents are required to operate software in virtual machines via keyboard, mouse, or terminal, and are assessed based on the resulting files and application states rather than their own descriptions.
In evaluations of 12 vision-language models, the average task score was only 34.3%. Claude Fable 5.1 led with 73.7%, followed by Claude Opus 5 at 57.9%, GPT-6 Astra at 57.4%, and GPT-5.6 sol at 50.1%. Performance varied significantly across disciplines: models generally performed well on statistical tasks, while only a few exceeded 20% on chemistry tasks. GPT-6 Astra achieved 69% on medical imaging tasks but performed poorly in chemistry.
The analysis also found that 61% of failed runs left no scorable artifacts and exhausted their step or time budgets. One model repeatedly clicked the same incorrect location 39 times due to coordinate mapping errors, ultimately failing the task. Additionally, the top-scoring model was not the most expensive to run, indicating that performance and cost are not simply linearly related. The team says OSWorld-Science provides a set of tasks and execution logs for continuously evaluating and comparing agents' scientific task performance, and plans to expand with more real scientific problems and cross-software AI workflows.
