Safe AI for students: what school leaders should require

"Safe" is easy for an AI vendor to claim and hard for a school to test. A content filter matters, but it tells you nothing about where a student's conversations are stored, who can read them, whether their essays end up training someone's model, or what happens when a student types something worrying late at night.

This guide is for heads of school, IT leads and safeguarding leads who need a definition of "safe" they can hold a vendor to. It draws on public guidance from Singapore, the UK, UNESCO and the US, and ends with 12 questions to put to any vendor, in writing, before a student logs in.

What does "safe AI for students" actually mean?

Safety belongs to the whole system: the model, the product built around it, the contract, and your school's own processes. A useful working definition has five layers.

1. What the AI will and won't do. Its stated purpose, the topics it declines, and whether it does students' thinking for them. A tool that writes the essay on request fails here.
2. What it knows about the student. Which personal data it collects, where that data goes, how long it stays, and whether it trains models.
3. Who can see what. What staff see of student conversations, what students are told, and who is alerted when something looks wrong.
4. How it fails. Jailbreaks, harmful or false outputs, and companion-style use, where a student starts treating the tool as a friend.
5. Who is accountable. Contracts, logs, risk assessments, incident response and a named person on the vendor's side.

Procurement needs answers on all five.

What does public guidance already expect?

No single document covers every school, but five public bodies cover most of the ground between them.

The UK Department for Education (DfE) has the most detailed product checklist of the five. Its Generative AI: product safety standards (first published January 2025, updated 19 January 2026) are written for suppliers and apply to England, but the DfE says schools may also find them helpful when judging products. Learner-facing tools are expected to filter harmful content reliably, log prompts and responses, alert the designated safeguarding lead to possible concerns, resist jailbreaking, and not use learners' personal data or work for model training without a lawful basis or the owner's permission. A January 2026 update added standards on cognitive development (hints before full answers, by default), emotional and social development, mental health and manipulation; one expects suppliers "not to anthropomorphise products".

Singapore's Personal Data Protection Commission (PDPC) sets a higher bar for children's data. Its Advisory Guidelines on the PDPA for Children's Personal Data in the Digital Environment (28 March 2024) name technology-aided learning as in scope and define a child as anyone under 18. Children's personal data is generally treated as sensitive and owed "a higher standard of protection". A 13- to 17-year-old may give valid consent if the policies are readily understandable to them, though in an education setting it may be more prudent to ask a parent; under 13, a parent or guardian must consent. The guidelines also expect data minimisation and encourage a data protection impact assessment (DPIA) before releasing a product children are likely to use. They are advisory: ask your data protection officer how they apply to your school.

Singapore's Ministry of Education (MOE) sets a benchmark for its own tools. Its AI in Education Framework (page last updated 10 September 2026) rests on four principles: Agency, Inclusivity, Fairness and Safety. In a parliamentary reply on 6 May 2026, MOE said its AI Learning Assistant in the Student Learning Space redirects students who drift off-topic or want direct answers, and that data from students' use of MOE-built tools is anonymised and "not used to train external AI models". When schools use commercial AI tools, they must check that the input data contains no personally identifiable information. This guidance is written for MOE schools and does not state that it binds independent or international schools, so treat it as a benchmark rather than a rule.

UNESCO calls for age limits and privacy protection. UNESCO's Guidance for generative AI in education and research (7 September 2023) asks for regulation that protects users' data privacy and proposes an age limit for independent conversations with generative AI platforms. UN News reported on the same day that the limit under consideration for classroom use was 13. UNESCO also notes that, without national regulation, institutions are largely unprepared to validate these tools.

The US Department of Education puts people first. Its Office of Educational Technology report Artificial Intelligence and the Future of Teaching and Learning (May 2023) calls for "humans in the loop" as a key criterion for educational use of AI, and says AI tools must respect data privacy and security. Where students use chatbots, it says teachers will need to monitor that use and offer human recourse when things go wrong.

Together they point the same way: collect less, train on nothing without permission, keep an adult able to see and step in, plan for failure, and put it in writing.

What should you ask every vendor? The 12-question checklist

Ask for written answers, ideally in the contract or its schedules; a demo is not an answer. The order matters: early questions can end a conversation quickly, and later ones are what you test in a pilot.

1. What is the tool for, and what will it refuse to do? Look for a stated purpose, age range and subject focus, and a clear position on doing students' work for them.
2. What personal data do you collect from students, and why each item? Ask for a data map that names every field and its purpose. Anything collected without a stated purpose is a red flag.
3. Where is student data stored and processed, and who are your subprocessors? Ask for hosting regions, the full subprocessor list (including the AI model provider), how cross-border transfers are safeguarded, and how you will be told when the list changes.
4. Is any student data or student work used to train or improve AI models? Include the AI model provider in the question. You want a contractual no, and to know what the provider keeps and for how long.
5. How long do you keep conversations and records, and how do we delete them? Ask for retention periods by type of data, deletion on request and at the end of the contract, and an export format so you are not locked in.
6. What age is the tool designed for, and how does consent work? Ask how age is established, who consents for students under 13 and for those aged 13 to 17, and what the vendor relies on the school to do. Then take the answer to your data protection officer.
7. How do you filter harmful content and resist misuse? Ask whether filtering works across languages, misspellings and images, how it adapts to age and special educational needs, and how the vendor tests for jailbreaks.
8. What happens when a student discloses harm or distress? Ask for the escalation path: what triggers an alert, who receives it, how quickly, what happens out of hours, and how your safeguarding lead's details are kept current. Ask to see a sample alert.
9. What can staff see, and what are students told? Ask which roles can read conversations, whether staff access is recorded in an audit log, and how students learn that their conversations are visible. Visibility should be role-based, logged and disclosed.
10. Does the tool present itself as a person or encourage reliance? Look for no claims of feelings or friendship, default time limits, and no flattery or design that rewards time spent over learning.
11. How do you test safety, and what happens when something goes wrong? Ask about testing with realistic users before each release, the vendor's DPIA and risk assessment, an incident response plan with a committed time to notify the school, and a complaints route.
12. What will you put in writing, and what can you not yet show? Ask for the data processing agreement, subprocessor list, DPIA, risk assessment and any security certifications with their dates. Candour about what is still in progress beats a row of badges with no documents behind them.

How should you weigh the answers?

• Treat "we're compliant" without documents as no answer.
• Watch where responsibility lands. Be wary of answers that pass it to the model provider or to teachers.
• Review as a team. Your safeguarding lead, IT lead and data protection officer will each catch different gaps.
• Pilot before students arrive. Use staff and test accounts to probe the filters, follow a test alert through to the right person, and request a deletion.
• Ask again when things change. A new model provider or feature can change every answer above.

Vendors' answers will also shape your own AI policy, including on assessed work.

Where does The Infinity Lab stand?

I lead the safety systems and school controls at The Infinity Lab, so these are the questions I expect schools to put to us. We're building The Infinity Lab to answer each of them in writing for every partner school. Read how we handle safety and student data today, or see how we approach AI in learning.

If you are weighing up AI for your students, register your school's interest and tell us which of these 12 questions matters most to you.