DrivingBench Test: GPT-6 Astra Is the Only Model to Complete a Real-Vehicle Course
DrivingBench, an independent research project, released results from a real-vehicle control test in which GPT-6 Astra, Claude Fable 5.1, Grok 4.6, and GPT-5.6 Sol each drove a 2022 Toyota Corolla along a cone-marked route in a parking lot. GPT-6 Astra was the only model to complete the full course.
On its second attempt, Astra covered 134.7 meters in 5 minutes and 22 seconds, consuming about 6.6 million tokens at an estimated cost of $7.74. The best scores for the other models were 45%, 11%, and 6%. In its first attempt, Astra drove 67.3 meters, completing about 49% of the route, before a safety operator intervened. After reviewing the failure in the same conversation, Astra concluded that it had aligned too early and accelerated to 1.5 m/s, leaving too little room for correction. On the second run, it kept speed below 0.8 m/s and used 100% steering requests in 20 of 24 motion commands.
Created by three researchers, DrivingBench aims to test whether general-purpose models can directly operate a real car. The test vehicle was fitted with a comma four device connected to the CAN bus via OBD-C, with low-level control built on the open-source openpilot system. Camera feeds and vehicle states were streamed to a computer and passed to the model through MCP tools. The model issued commands via three tools: observing the surroundings, setting direction and speed, and stopping immediately; final steering, acceleration, and braking were executed by openpilot.
The team said Astra's performance showed notable in-context learning: the model's parameters were not retrained, but it adjusted its strategy based on information from the first run. However, the test took place in a closed parking lot with strict speed limits, and each model completed only one continuous-context session, which the researchers acknowledged is far from sufficient to prove stable driving capability. They also found that Astra initially sometimes refused to control the vehicle for safety reasons; after renaming the MCP tool "DrivingBench Sandbox," refusals decreased noticeably, though the exact cause remains unclear.
