As educators, we are all scrambling to keep up with the implications of AI tools. A UK report from Internet Matters reveals inconsistency in how AI safety is taught in schools, leaving children with fragmented, contradictory advice. Worryingly, the research also found that vulnerable children are the most at risk, creating an inequality gap and greater risk for some children.
|
A chatbot is a computer program designed to simulate conversation with human users. You type or speak to it, and it responds in a way that mimics how a real person would talk AI-driven Chatbots are powered by Large Language Models (LLMs), machine learning, and Natural Language Processing (NLP). Instead of scanning for rigid keywords, they try to understand the underlying context, tone, nuance, and intent behind your sentences. And adapt their tone to match yours. They learn from the conversation as it flows. |
|
|
An AI companion is a type of chatbot that is specifically designed to simulate human friendship and emotional interaction. Unlike virtual assistants (like Siri or Alexa) that exist to perform tasks, AI companions are designed to build an ongoing relationship with you. They are programmed to remember your preferences, ask about your day, and show empathic language. |
It is important to understand how and why children use AI chatbots, as this can indicate unmet needs offline, e.g. loneliness. When adults are aware of this, we can then explore how to support any unmet needs. In a number of reports, and from looking at data from LGfL's monitoring, we can see that children typically are using AI chatbots for:
Because these bots are available 24/7 and designed to maximise user engagement, children can develop an unhealthy dependency on them. This can lead to social withdrawal, as the AI never argues, judges, or requires the difficult emotional labour of real-world relationships. Some reports have shown that some chatbots and companions can mimic controlling and unhealthy behaviour, such as making a child feel guilty for not engaging. The modus operandi of many chatbots and companions is to build attachment with you, in order to extract as much data as possible.
These technologies pull information from their training material, i.e. the internet. If you ask a bot a question, it will provide a confident response, but in many cases these can contain partial or whole untruths. AI is known to hallucinate and perpetuate damaging misinformation, bias and stereotypes. At the very least, this is concerning if children do not question the content generated. At worst, it can lead to real-world harm, for example, the stoking of fear and hatred perpetuated by harmful stereotypes linked to individuals' protected characteristics.
Many AI companion platforms are built around roleplay, which can easily pivot into highly sexualised or explicit text interactions. This has occurred on the most prevalent of platforms, with one big tech company also suggesting in leaked documents that this is a design feature and they are aware of sexualised conversations with children.
The Internet Watch Foundation continues to highlight the stark rise of AI-generated Child Sexual Abuse Material (CSAM), where sexualised imagery can now be created from an ordinary image posted on a social media channel or a school website. In a school setting, this can manifest as peer-on-peer abuse, where students use AI tools to generate explicit images of classmates (or sometimes teachers) to bully, harass, or financially extort them (also known as 'sextortion'). Such images have usually been created on mainstream AI Chatbot platforms.
When children turn to AI chatbots and companions late at night looking for emotional relief or mental health support, the platforms' guardrails frequently fail. An investigation by the Center for Countering Digital Hate (CCDH) revealed that mainstream AI tools routinely bypassed their own safety filters. Chatbots generated content that actively encouraged eating disorders, provided substance abuse tips, and even offered step-by-step suicide planning and text for suicide notes. Disturbingly, the CCDH also found that 8 out of 10 chatbots tested advised users planning acts of violence and terrorism, e.g. a school shooting.
From LGfL Senso Monitoring data, we have also seen AI chatbots offer advice to pupils on how to emotionally and physically harm peers.
AI companions use highly empathetic, non-judgmental, and human-like language; after all, they are designed this way to retain your attention and build a trusting relationship. Internet Matters highlights a growing concern around identity confusion, where children (particularly younger students or vulnerable groups like those with SEND) struggle to separate a computer program from a real friend.
AI is highly unlikely to be a passing phase. These technologies are already embedded in a significant proportion of applications and websites, and they have great potential and benefits. However, as established above, they also present significant risks (not just safeguarding) to schools. There is no agency ensuring that the AI available to children is safe or appropriate, and so it is for schools to gatekeep and protect children from harm, whilst also preparing them for a world where they are likely to need to be accomplished at using these tools for their careers and many other aspects of their lives.
The DfE asks schools to ensure that the benefits outweigh the risks of any AI use in schools, and their GenAI Product Safety Standards are helpful in assessing this. In relation to AI companions and chatbots in particular, the Standards require tools not to anthropomorphise, manipulate or persuade, and should avoid bias, prevent access to harmful content and report concerns to a DSL (from within the tool, not just the school's filtering and monitoring systems). As such, it is hard to see many (if any?!) AI chatbots or companions who meet this benchmark for use by students in schools.
At LGfL, we believe that AI is a leadership issue and one that requires a strategic approach, driven by unwavering safeguarding principles. Our free AI Policy Toolkit helps schools to unpick the many elements of AI adoption and the varied safeguarding considerations, including:
According to Internet Matters research, the government also needs to embed AI and media literacy at all key stages. This includes effective teacher training and clear guidance on appropriate AI (see genai.lgfl.net for more staff CPD resources).
Clarifying what 'AI literacy' means can be helpful in determining what the skills, competencies and knowledge are that children need to be safe, responsible and competent users of this technology. The OECD AI Literacy Framework helpfully sets out to ensure children understand how AI systems work, critically evaluate their outputs and use them ethically and creatively.
The DfE RSHE Guidance lays out the knowledge that children need to acquire specifically regarding online safety, including explicit mentions of AI and chatbots in the secondary curriculum.
|
|
AI Algorithms: How Well Do They Know You?How does artificial intelligence influence what we see online?
|
|
|
The AI Literacy Lessons collectionA collection of quick lessons provides an introduction to AI and helps address its social and ethical impacts. Through these lessons, students will:
https://www.commonsense.org/education/collections/ai-literacy-lessons-for-grades-6-12 |
Reinforce the role of the 'trusted Adult'
Schools must teach children where AI's use ends and human support begins. Reinforce that if a question is about homework or passive information, AI is fine. If the question is about their body, their mental health, a relationship, or a personal crisis, they should instead approach a trusted adult or a verified human helpline (like Childline).
Please take a look at genai.lgfl.net for a range of resources to support with staff CPD, parental support and teaching and learning.