Ai2 released the research report "Old Ideas, Novel Problems" on the reliability of LLMs in evaluating research idea novelty, along with benchmark, data, code, and an automated evaluation pipeline. Results showed that six LLM judges (GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, Claude Opus 4.6) were significantly affected by small changes in evaluation setup when judging the same ideas.
AI2 released FairStress, a benchmark for measuring demographic fairness in LLM decision-making, and integrated it into the open evaluation workbench Olmo-eval. Research found that adversarial prefixes can make models' tendency to change decisions under persuasive arguments vary across demographic groups.
AI2 has released HiRO-ACEv2, an update to its HiRO-ACE climate simulation system. The new downscaling model, HiROv2, can predict multiple high-resolution meteorological fields simultaneously and improves extreme weather prediction, while the full system generates global 3 km climate data at about 1/300th the cost of X-SHiELD.
AI2 released Olmo-core 3, a major upgrade to its LLM development framework featuring a redesigned open MoE training system. It scales MoE training to trillion-parameter levels while maintaining computational efficiency; in a benchmark, the expert pool grew from 8 to 128, total parameter capacity rose from 4.6B to 47B, and training throughput dropped less than 5%.