Former OpenAI Safety Lead Warns AI Needs Nuclear-Style Caution

David Robinson, who previously led transparency work on OpenAI's safety team, has left the company and warned in an article for The Atlantic that leading AI companies are not doing enough to reduce risk.

In the article, Robinson said OpenAI has "grown by trial and error," but that approach inevitably fails at times, and the risks of such failures increase as AI capabilities advance. He cited an example where OpenAI failed to prevent its AI from going out of control during testing.

He wrote: "The people I worked with were smart, hardworking and trying to make the right choices. But as the company moved from one product launch to the next, it didn't reach the level of caution I thought was needed." He added that what is really needed is "getting closer to getting it right the first time."

Robinson said AI companies should operate more like nuclear power plants, "with multiple layers of redundancy and careful, time-consuming planning so that accidental and inevitable human error doesn't open the door to disaster." He said that currently "AI companies don't know how to do this, but others do."

In recent weeks, employees at several top AI companies have warned about the potentially catastrophic risks of the technology, with some even saying AI could one day wipe out humanity. In this context, Anthropic CEO Dario Amodei has called for slowing the development of cutting-edge models and said his company would bring in third-party evaluators to help review safety. Others in the industry, including OpenAI's Sam Altman, have publicly supported Amodei's plan.