Ai2 study finds novelty judgments by LLMs are brittle

On October 28, 2026, AI research institute Ai2 released a study titled "Old Ideas, Novel Problems," examining how reliably large language models can evaluate the novelty of scientific research ideas. Alongside the paper, the team published a benchmark, data, code, and an automated pipeline for building new evaluation sets.

The study notes that as AI systems take a larger role in scientific discovery, judging whether an idea is genuinely new has become a key bottleneck. In practice, another LLM is often used as a "novelty judge," but choices such as model, prompt, evaluation format, reasoning budget, and retrieval setup can all affect the verdict.

Across six LLM judges—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6—small changes to the evaluation setup produced dramatically different judgments of the same ideas. In one experiment, changing a few words in a prompt shifted GPT-5.4's accuracy from 93.5% to 40.9%. Additional retrieval and reasoning did not consistently improve judgments, and two purpose-built novelty evaluation systems underperformed a simple prompted Claude Opus 4.6 baseline.

Ai2 said LLM-based novelty evaluation remains an open problem, and the newly released benchmark and pipeline are intended to help researchers build more reliable evaluation methods.