OpenAI launches CSAM classifier and ChatGPT safety warnings
OpenAI says its new classifier will help vetted organisations detect potential CSAM, while ChatGPT adds warnings and 24-hour account pauses.
OpenAI has released a new image classifier designed to help vetted organisations detect potential child sexual abuse material (CSAM), including material generated by artificial intelligence. The company says the tool analyses visual signs of abuse and returns scores that help safety teams decide which images should be reviewed first. It is available at no cost through the OpenAI API to eligible organisations.1
The classifier is intended to help teams find unfamiliar images, rather than simply match material already known to them. OpenAI says it developed and evaluated the system with assistance from the National Center for Missing & Exploited Children, and is working with the Tech Coalition and ROOST on wider access and collaboration.1
The early results
OpenAI says an internal evaluation found that the classifier could reduce the number of images sent for review by 48% while identifying roughly the same number of abusive images as its existing tool. Images flagged by the new system were about twice as likely to contain abuse, according to the company. Those are OpenAI's early test results, not an independent benchmark, and the announcement does not publish a complete external evaluation dataset.1
The company says organisations can request access through its intake form, with eligibility and deployment handled through a vetting process. OpenAI has also published a model card explaining the classifier and its intended use.2
ChatGPT warnings and account pauses
OpenAI is pairing the classifier launch with new ChatGPT interventions. When its systems detect attempts to find abusive material, ChatGPT will display a warning explaining that sexualising children is harmful and illegal, remove the flagged conversation and pause access for 24 hours. Repeated violations may lead to longer restrictions, account closure or reports to authorities, according to OpenAI.1
The warning also points people towards specialist support. OpenAI says the approach was developed with input from child-safety organisations and academic advisers, while its separate developer guidance sets out processes for prevention, detection, reporting and review.1
Our opinion
The important part of this announcement is not a flashy AI feature but the division of labour around it. A classifier can prioritise work for human reviewers, but OpenAI's own figures are early company-reported results rather than proof that the system will perform equally well across different services and image types. The useful test will be whether vetted organisations can deploy it responsibly, understand its limitations and keep people accountable for the final decisions.