OpenAI’s ‘Project Lily’ Exposed: Human Reviewers Read ChatGPT Conversations

404 Media has published an in-depth report revealing an internal OpenAI project codenamed “Project Lily,” in which human reviewers examine anonymized real user chat logs to evaluate the quality of ChatGPT responses. The outlet said it obtained operation guidelines, Slack channel records, sample conversations, and a scoring system used for grading. Reviewers, called “prompt reviewers,” assess whether answers stay on topic and flag issues such as excessive “AI-like” phrasing, condescending tone, emojis, or sycophantic language. One reviewer described the work as “very repetitive,” with reported hourly pay exceeding $50.

OpenAI acknowledged to 404 Media that its filtering process could miss some personal data, particularly in shorter conversations. The report also noted that many users explicitly ask ChatGPT to keep conversations confidential, and the versions given to reviewers include a “user memory summary” that may contain interests, context, and location information. The project does not check factual accuracy beyond obvious errors and is separate from safety reviews aimed at detecting harmful intent.

After the report was published, OpenAI updated its help page to clarify how users can opt out of data collection, but the revised documentation still does not mention that human workers may read conversations. 404 Media noted that similar human review practices are common across AI services, including Google’s Gemini and Anthropic, while Perplexity’s privacy policy has not clearly stated whether employees can access chat logs.