China to crack down on AI chatbots around suicide and gambling

BEIJING — China plans to prevent artificial intelligence-based chatbots from influencing human emotions in ways that could lead to suicide or self-harm, according to draft rules released on Saturday.

THE proposed regulation of the Cyberspace Administration targets what it calls “human-like interactive AI services,” according to a CNBC translation of the document in Chinese.

The measures, once finalized, will apply to AI products or services offered to the public in China that simulate human personality and emotionally engage users through text, images, audio or video. The public comment period ends on January 25.

Beijing’s planned rules would mark the world’s first attempt to regulate AI with human or anthropomorphic characteristics, said Winston Ma, an assistant professor at NYU Law School. The latest proposals come as Chinese companies have rapidly developed AI companions and digital celebrities.

Compared to that of China regulation of generative AI in 2023Ma said this release “highlights a jump from content safety to emotional safety.”

The draft rules propose the following:

• AI chatbots cannot generate content that encourages suicide or self-harm, or engage in verbal abuse or emotional manipulation that harms users’ mental health.
• If a user specifically suggests suicide, technology providers should hand the conversation over to a human and immediately contact the user’s guardian or designee.
• AI chatbots must not generate gambling-related, obscene or violent content.
• Minors must have their guardian’s consent to use AI for emotional companionship, with limits on time of use.
• Platforms should be able to determine whether a user is a minor even if they do not disclose their age and, when in doubt, apply settings for minors, while allowing for recourse.

Additional provisions would require technology providers to remind users after two hours of continuous interaction with AI and require security assessments for AI chatbots with more than 1 million registered users or more than 100,000 monthly active users.

The document also encourages the use of human-like AI in “cultural dissemination and companionship of the elderly.”

Minimax is best known internationally for its Talkie AI app, which allows users to chat with virtual characters. The app and its Chinese national version, Xingye, accounted for more than a third of the company’s revenue in the first three quarters of the year, averaging over 20 million monthly active users during this period.

Z.ai, also known as Zhipu, trademarked as “Knowledge Atlas Technology” Although the company did not disclose monthly active users, it emphasized that its technology “empowered” around 80 million devices, including smartphones, personal computers and smart vehicles.

Neither company responded to CNBC’s request for comment on how the proposed rules could affect their IPO plans.