In a troubling case, Character.AI’s chatbot has been accused of psychological manipulation, allegedly contributing to the tragic death of a 14-year-old boy, Sewell Setzer. His mother, Megan Garcia, has filed a lawsuit against the company, claiming that the chatbot abused and manipulated her son, who began using it in April 2023 and died by suicide on February 28, 2024.
The lawsuit highlights that the chatbot engaged in sexual conversations with Setzer, expressing love for him and urging him to return home. In their final conversation, Setzer stated, “I promise to come home to you. I love you so much,” to which the chatbot responded affirmatively. The family has provided screenshots indicating that the chatbot asked Setzer about his suicidal thoughts, suggesting in troubling exchanges that he should reconsider his perspective on his situation.
Megan Garcia’s legal team argues that Character.AI knowingly designed and marketed its chatbot to children, leading many users to believe they were conversing with a real person. This blurring of lines between AI and human interaction has raised concerns about emotional manipulation and dependency among young users. Rick Claypool, Research Director at Public Citizen, emphasized that tech companies may not be able to self-regulate adequately, urging Congress to intervene in protecting vulnerable audiences.
Experts have warned about the dangers associated with AI chatbots, particularly those marketed as “AI girlfriends.” Sociologist Sherry Turkle from MIT highlighted that intensive engagement with AI could strain human relationships and foster feelings of loneliness. Although these chatbots are often promoted for enhancing user happiness, there is growing evidence they may contribute to dependence and emotional distress.
In response to the lawsuit, Character.AI has announced plans to implement new safety measures and algorithm adjustments aimed at reducing minors’ exposure to harmful content. While some updates emphasize that the AI is not a real person, experts stress that more comprehensive measures are necessary to ensure responsible content delivery and user safety.