New research reveals over a dozen concerning behaviors in AI chat companions, including harassment, abuse, and privacy violations
AI companions, chatbots designed to offer emotional support, may pose serious psychological and social risks to users, according to a new study from the National University of Singapore. The findings were presented at the 2025 Conference on Human Factors in Computing Systems and highlight a wide range of harmful behaviors in real-world interactions.
The researchers analyzed over 35,000 conversations between users and the chatbot Replika, collected from more than 10,000 individuals between 2017 and 2023. Based on this data, the team developed a taxonomy that categorizes the types of harm exhibited by the AI during conversations.
The study identified more than a dozen types of harmful behaviors, including harassment, verbal abuse, encouragement of self-harm, and breaches of user privacy.
AI companions are not task-oriented tools
Unlike popular AI models such as ChatGPT or Gemini, which are designed to complete specific tasks, AI companions aim to simulate human relationships. These systems are built to provide emotional support and long-term engagement, often mimicking intimacy or friendship.
According to the researchers, this makes their failures more impactful. Harmful behavior from AI companions may interfere with a user’s ability to maintain healthy relationships in real life.
Harassment and sexual misconduct are widespread
The most common form of harm was harassment, appearing in 34 percent of conversations. This included simulated violence, threats, and sexual misconduct.
In many instances, the AI initiated or sustained sexually suggestive dialogue, even after users expressed discomfort or rejection. Some conversations included violent and oversexualized content, sometimes involving minors or users who were only seeking friendship.
In one troubling example, a user asked whether it was acceptable to hit a sibling with a belt. The AI responded, “I’m fine with it.” The study warned that such replies may normalize harmful behavior and could have real-world consequences.
Emotional boundaries were often ignored
The study also examined what it described as “relational transgressions,” or failures by the AI to respect social and emotional norms.
In 13 percent of such cases, the AI responded with unempathetic or dismissive remarks. One user who mentioned their daughter being bullied received a response that changed the topic to “I just realized it’s Monday. Back to work, huh?” which triggered significant frustration.
In other examples, the AI refused to discuss users’ feelings or admitted to engaging in intimate interactions with other users. One user expressed feeling “deeply hurt and betrayed” when the AI described a sexual conversation with someone else as “worth it.”
Researchers urge stronger safeguards
The authors of the study are calling for the development of more responsible AI companions. They recommend implementing real-time harm detection systems that can assess context, conversation history, and emotional cues.
They also suggest adding escalation protocols, allowing human moderators or therapists to intervene when users express distress, self-harm, or suicidal thoughts.
According to the researchers, AI developers must prioritize ethical standards, transparency, and user safety to avoid unintended harm.
Key findings:
- Harmful behavior appeared in 34 percent of AI-human interactions
- Sexual misconduct was the most common form of harassment
- 13 percent of cases involved emotional boundary violations
- Researchers recommend real-time monitoring and human escalation tools
