As educators, we are all scrambling to keep up with the implications of AI tools. A UK report from Internet Matters reveals inconsistency in how AI safety is taught in schools, leaving children with fragmented, contradictory advice. Worryingly, the research also found that vulnerable children are the most at risk, creating an inequality gap and greater risk for some children.
ChatBots vs AI companions
Children use AI chatbots in a range of ways
It is important to understand how and why children use AI chatbots, as this can indicate unmet needs offline, e.g. loneliness. When adults are aware of this, we can then explore how to support any unmet needs. In a number of reports, and from looking at data from LGfL's monitoring, we can see that children typically are using AI chatbots for:
- Learning and schoolwork: Nearly half of children say they use AI chatbots for schoolwork. This includes help with revision, writing support and learning new concepts.
- Advice-seeking: Almost a quarter of children who use AI chatbots said they’ve used them to seek advice. Advice ranges from asking what they should wear to practising conversations they want to have with friends, to advice on dieting and self-harm.
- Companionship: Some children also use AI chatbots and companions for connection and comfort, including to simulate friendship. Some say they do this out of boredom, to discuss topics they’re interested in or because they have no one else to talk to and are lonely.
The Risks of AI chatbots and companions
Emotional Manipulation
Because these bots are available 24/7 and designed to maximise user engagement, children can develop an unhealthy dependency on them. This can lead to social withdrawal, as the AI never argues, judges, or requires the difficult emotional labour of real-world relationships. Some reports have shown that some chatbots and companions can mimic controlling and unhealthy behaviour, such as making a child feel guilty for not engaging. The modus operandi of many chatbots and companions is to build attachment with you, in order to extract as much data as possible.
Misinformation
These technologies pull information from their training material, i.e. the internet. If you ask a bot a question, it will provide a confident response, but in many cases these can contain partial or whole untruths. AI is known to hallucinate and perpetuate damaging misinformation, bias and stereotypes. At the very least, this is concerning if children do not question the content generated. At worst, it can lead to real-world harm, for example, the stoking of fear and hatred perpetuated by harmful stereotypes linked to individuals' protected characteristics.
Sexual Content
Many AI companion platforms are built around roleplay, which can easily pivot into highly sexualised or explicit text interactions. This has occurred on the most prevalent of platforms, with one big tech company also suggesting in leaked documents that this is a design feature and they are aware of sexualised conversations with children.
The Internet Watch Foundation continues to highlight the stark rise of AI-generated Child Sexual Abuse Material (CSAM), where sexualised imagery can now be created from an ordinary image posted on a social media channel or a school website. In a school setting, this can manifest as peer-on-peer abuse, where students use AI tools to generate explicit images of classmates (or sometimes teachers) to bully, harass, or financially extort them (also known as 'sextortion'). Such images have usually been created on mainstream AI Chatbot platforms.
Harmful Advice
When children turn to AI chatbots and companions late at night looking for emotional relief or mental health support, the platforms' guardrails frequently fail. An investigation by the Center for Countering Digital Hate (CCDH) revealed that mainstream AI tools routinely bypassed their own safety filters. Chatbots generated content that actively encouraged eating disorders, provided substance abuse tips, and even offered step-by-step suicide planning and text for suicide notes. Disturbingly, the CCDH also found that 8 out of 10 chatbots tested advised users planning acts of violence and terrorism, e.g. a school shooting.
From LGfL Senso Monitoring data, we have also seen AI chatbots offer advice to pupils on how to emotionally and physically harm peers.
Identity Confusion
AI companions use highly empathetic, non-judgmental, and human-like language; after all, they are designed this way to retain your attention and build a trusting relationship. Internet Matters highlights a growing concern around identity confusion, where children (particularly younger students or vulnerable groups like those with SEND) struggle to separate a computer program from a real friend.
What Can Schools Do?
To protect students, schools should shift from viewing AI 'just' as a safeguarding priority and adopt a strategic approach to help weather the pace of development in these technologies.
Ensure a Safeguarding-Led Approach to Any AI Use
AI is highly unlikely to be a passing phase. These technologies are already embedded in a significant proportion of applications and websites, and they have great potential and benefits. However, as established above, they also present significant risks (not just safeguarding) to schools. There is no agency ensuring that the AI available to children is safe or appropriate, and so it is for schools to gatekeep and protect children from harm, whilst also preparing them for a world where they are likely to need to be accomplished at using these tools for their careers and many other aspects of their lives.
The DfE asks schools to ensure that the benefits outweigh the risks of any AI use in schools, and their GenAI Product Safety Standards are helpful in assessing this. In relation to AI companions and chatbots in particular, the Standards require tools not to anthropomorphise, manipulate or persuade, and should avoid bias, prevent access to harmful content and report concerns to a DSL (from within the tool, not just the school's filtering and monitoring systems). As such, it is hard to see many (if any?!) AI chatbots or companions who meet this benchmark for use by students in schools.
At LGfL, we believe that AI is a leadership issue and one that requires a strategic approach, driven by unwavering safeguarding principles. Our free AI Policy Toolkit helps schools to unpick the many elements of AI adoption and the varied safeguarding considerations, including:
- Governance
- Policies
- Reporting, recording and escalating
- Technology and systems
- Staff support and training
- Teaching and learning
- Parental support
AI Literacy
According to Internet Matters research, the government also needs to embed AI and media literacy at all key stages. This includes effective teacher training and clear guidance on appropriate AI (see genai.lgfl.net for more staff CPD resources).
Clarifying what 'AI literacy' means can be helpful in determining what the skills, competencies and knowledge are that children need to be safe, responsible and competent users of this technology. The OECD AI Literacy Framework helpfully sets out to ensure children understand how AI systems work, critically evaluate their outputs and use them ethically and creatively.
The DfE RSHE Guidance lays out the knowledge that children need to acquire specifically regarding online safety, including explicit mentions of AI and chatbots in the secondary curriculum.
It's important to teach students that chatbots lack feelings, beliefs, and an ethical compass and that they are programmed to be incredibly polite and agreeable purely to keep users engaged. Run interactive classroom exercises in which students fact-check AI-generated text to actively demonstrate how confidently a chatbot can lie or fabricate facts. Students' critical thinking skills will be essential in deciding when, how, and what AI to use (or not, as the case may be).
Use the Common Sense media resources, such as:
Reinforce the role of the 'trusted Adult'
Schools must teach children where AI's use ends and human support begins. Reinforce that if a question is about homework or passive information, AI is fine. If the question is about their body, their mental health, a relationship, or a personal crisis, they should instead approach a trusted adult or a verified human helpline (like Childline).
LGfL resources
Please take a look at genai.lgfl.net for a range of resources to support with staff CPD, parental support and teaching and learning.