Amazon discovered a ‘high volume’ of CSAM in its AI training data but isn’t saying where it came from

The Nationwide Middle for Lacking and Exploited Kids mentioned it obtained greater than 1 million studies of AI-related youngster sexual abuse materials (CSAM) in 2025. The “overwhelming majority” of that content material was reported by Amazon, which discovered the fabric in its coaching information, in accordance with an investigation by Bloomberg. As well as, Amazon mentioned solely that it obtained the inappropriate content material from exterior sources used to coach its AI companies and claimed it couldn’t present any additional particulars about the place the CSAM got here from.

“That is actually an outlier,” Fallon McNulty, government director of NCMEC’s CyberTipline, instructed Bloomberg. The CyberTipline is the place many kinds of US-based corporations are legally required to report suspected CSAM. “Having such a excessive quantity are available all year long begs a variety of questions on the place the information is coming from, and what safeguards have been put in place.” She added that apart from Amazon, the AI-related studies the group obtained from different corporations final 12 months included actionable information that it may go alongside to legislation enforcement for subsequent steps. Since Amazon isn’t disclosing sources, McNulty mentioned its studies have proved “inactionable.”

“We take a intentionally cautious strategy to scanning basis mannequin coaching information, together with information from the general public internet, to determine and take away identified [child sexual abuse material] and defend our clients,” an Amazon consultant mentioned in an announcement to Bloomberg. The spokesperson additionally mentioned that Amazon aimed to over-report its figures to NCMEC to be able to keep away from lacking any instances. The corporate mentioned that it eliminated the suspected CSAM content material earlier than feeding coaching information into its AI fashions.

Security questions for minors have emerged as a important concern for the substitute intelligence trade in current months. CSAM has skyrocketed in NCMEC’s data; in contrast with the greater than 1 million AI-related studies the group obtained final 12 months, the 2024 whole was 67,000 studies whereas 2023 solely noticed 4,700 studies.

Along with points resembling abusive content material getting used to coach fashions, AI chatbots have additionally been implicated in a number of harmful or tragic instances involving younger customers. OpenAI and Character.AI have each been sued after youngsters deliberate their suicides with these corporations’ platforms. Meta can be being sued for alleged failures to guard teen customers from sexually specific conversations with chatbots.

Replace: Jan 30, 4:00am ET:

An Amazon spokesperson has shared the next statements with Engadget:

“Amazon is dedicated to stopping CSAM throughout all of its companies, and we aren’t conscious of any situations of our fashions producing CSAM. In accordance with our commitments to accountable AI and the Generative AI Ideas to Forestall Youngster Abuse, we take a intentionally cautious strategy to scanning basis mannequin coaching information, together with information from the general public internet, to determine and take away identified CSAM and defend our clients. Whereas our proactive safeguards can not present the identical element in NCMEC studies as consumer-facing instruments, we stand by our dedication to accountable AI and can proceed our work to forestall CSAM.”

“We deliberately use an over-inclusive threshold for scanning, which yields a excessive share of false positives.”

“After we arrange this reporting channel in 2024, we knowledgeable NCMEC that we might not have enough info to create actionable studies, due to the third-party nature of the scanned information. The separate channel ensures that these studies wouldn’t dilute the efficacy of our different reporting channels. Due to how this information is sourced, we do not have the information that contains an actionable report.”