Meta creates monitoring tool for kids using its chatbot to discuss self-harm — and alerts their parents

Meta is launching a new feature designed to prioritize child safety within its AI chatbots.

The update will notify parents via Instagram’s supervision tools if their teenager discusses suicide or self-harm with Meta AI.

These protections allow Meta AI to direct teenagers to a crisis helpline if they mention suicide or self-harm in messages, while also encouraging them to reach out to a parent or a trusted adult.

Meta will send an alert to parents if a chat suggests their child is at risk. Experts have collaborated with Meta to identify signals that warrant an alert, flagging clear or subtle references to self-harm. A dedicated AI system was specifically developed to identify these sensitive conversations.

Once the technology flags a chat, it undergoes a manual review before an alert is sent. If the intent is ambiguous, Meta stated it will err on the side of caution and notify the parent. The company noted that it would rather issue a false alarm than miss a potential risk, citing this as the appropriate starting point.

In a statement, Meta — Facebook’s parent company — said, “We understand how distressing these alerts may be for a parent to receive. That’s why, as we continue to improve our detection, all chats flagged by our AI will be manually reviewed before an alert is sent.”

Instagram already sends alerts to supervising parents if a teenager uses Instagram to search for suicide or self-harm content, which started in February. These AI alerts are now live for the parents who use Instagram parental supervision in the US, UK, Australia and Canada and will be available globally by the end of the year. 

Meta also stated that they are building a plan to contact emergency services if a conversation suggests that the user is at imminent risk of suicide. Meta currently looks across Facebook and Instagram for posts suggesting a credible risk of suicide and alerts emergency services. 

Meta has already made over 19,000 referrals globally and helped first responders perform wellness checks on users who seemed at risk. 

This feature was developed with the help of more than 75 clinicians who specialize in teen mental health. They reviewed how AI responds to hundreds of prompts and worked with the AI Wellbeing Expert Council, Suicide and Self-Harm Advisory Group and Youth Advisors to provide feedback on appropriate responses and what was working, as well as improvements. 

The feedback included making sure Meta AI interacts with the teen in a way that acknowledges their feelings and doesn’t shut the conversation down too abruptly. 

Parents will also be sent expert resources that can help them approach conversations about suicide and self-harm with teenagers. 

Meta is implementing various conversation tools on Teen Accounts, which are automatically placed into a 13+ content setting, designed to give age-appropriate responses and not comply with sexual or romantic conversations, as well as not provide any information that would direct teens towards alcohol. 

In October, Instagram also released a Limited Content feature as a stricter content setting for parents who want restrictive posts on their child’s feeds. This will now also apply to AI experiences, meaning Meta AI will decline to respond to certain topics to reduce the chance of inappropriate conversation. 

These announcements are all following the UK government’s announced ban on social media for under-16-year-olds. The ban is expected in spring 2027, which will cover Snapchat, TikTok, YouTube, Instagram, Facebook and X. An opt-out social media curfew, which will disable “addictive features” between midnight and 6 a.m., has been announced in the UK as well.