Arena Chess Test: Claude Opus 5.5 Climbs to 1760 Elo While GPT-6 Astra Slides to 1400

Peter Gostev, head of AI capability at Arena, published on October 5, 2026, the results of a self-learning chess experiment in which Claude Opus 5.5 and GPT-6 Astra each played 200 games against Stockfish without any parameter updates, relying instead on their own notes to carry experience across games.

Stockfish, the strongest open-source chess engine, was split into three difficulty levels: Skill 0, a 1800-rated setting, and full strength. Models were allowed to choose their opponents, with one rule: they could not call a chess engine to play for them. Opus 5.5 stayed in the 1450-1550 range for the first 75 games, then climbed to about 1790 by game 150 and settled at 1760 after game 194. GPT-6 Astra followed the opposite path, peaking at 1810 in game 29, falling to about 1680 by game 100, and ending around 1400 after 200 games. Astra lost all 68 games it played against full-strength Stockfish.

Gostev noted that Elo estimates were based on the latest 50 games against the Skill 0 and 1800-rated Stockfish levels, with full-strength games excluded from the calculation. He described Opus 5.5's performance as remarkable and said he had checked that it did not cheat by calling a chess engine. For Astra's decline, he offered a tentative explanation: the model's 272k-token native context window in this Codex configuration may have struggled as notes and files accumulated, though controlled experiments are still needed to confirm.

The test was designed to see whether large language models can improve in actual play by reflecting on notes and past games without changing weights. GPT-6.1 Sol and Claude Fable 5.1 have since been added to the testbed, using a 828K-token context window and another configuration respectively, though they have only played a small number of games so far.