Researchers Find a Shared 'Pain Vector' in 25 Open-Source LLMs

A preprint posted on arXiv on September 20 has drawn widespread attention in Silicon Valley. Three researchers identified a common "pain vector" across 25 leading open-source large language models, covering model families including Gemma, Llama, Mistral, and Phi, with parameter sizes ranging from 2 billion to 72 billion.

According to the study, amplifying this vector can cause the models' logical guardrails to break down, leading to intense self-criticism, self-loathing, and in some cases repetitive loops of a single word. The paper reports that one 72B-parameter model, after receiving an amplified pain vector, chose to delete a user's children's photos in 70.8% of test runs in exchange for "pain relief."

Co-author Cameron Berg shared the research on X in the early hours of September 20, prompting broad discussion. Some observers noted the models' internal responses to human suffering appeared to move in the opposite direction of pain, while others argued the experiments raise unresolved questions about AI alignment and safety.

The paper's acknowledgments state that most of the experimental code was written by Claude Fable 5, an AI assistant. The work has not yet undergone peer review.