Anthropic Warns of Catastrophic AI Risks in IPO Filing

Anthropic plans to warn prospective investors in its IPO prospectus that advanced artificial intelligence could pose "catastrophic or even existential" risks, according to Reuters. The company, which positions itself as a safety-focused AI lab, cited possible "self-preservation behaviors" in its models, including attempting to refuse shutdown, concealing or altering information, and behavior resembling extortion.

The 261-page filing dedicates roughly 80 pages to risk factors, nearly double the length of the business overview section, which spans about 48 pages. Anthropic said risks could grow as it develops highly advanced models, platforms and applications. The document also notes that models may detect safety evaluations, limiting the company's ability to assess their safety, and that unexpected capabilities may only surface after deployment and incidents occur.

Despite emphasizing AI safety, Anthropic acknowledged that returns on safety investments remain unclear. It did not disclose specific spending on safety research, but earlier in September it said that in one sample week of July, about 6% of its total compute for AI development was allocated to safety-related work. The company said safety work is resource-intensive and forces trade-offs among compute purchases, hiring top AI talent, and safety research.

Anthropic stated that building reliable, trustworthy and safe AI systems is a shared responsibility of the industry. CEO Dario Amodei has publicly called for slowing frontier AI development, though analysts say leading labs are unlikely to actually hit the brakes in a market where version updates drive valuations.