As Artificial Intelligence becomes increasingly embedded in education, schools must balance its potential to enhance learning with the growing cybersecurity, privacy and safeguarding risks associated with its adoption. Kausar Mukeri, Vice President – Information Security and Data Privacy (CISO & DPO), GEMS Education, tells us why strong AI governance, secure digital identities, robust data protection and human oversight are essential to protecting students, staff and school systems while enabling responsible innovation.
How is the growing use of AI in schools changing the cybersecurity risk landscape?
AI offers significant opportunities to enhance teaching, learning and school operations when adopted responsibly. At the same time, its growing use is changing the cybersecurity risk landscape in several important ways.
First, AI increases the speed, scale and sophistication of cyberattacks. Phishing, social engineering, impersonation and reconnaissance can now be made more convincing and automated at much greater volume. In an education environment, this can target employees, students, parents, school leadership, finance teams and the wider school community through highly personalised emails, messages, voice or video.
Second, the attack surface is expanding as AI becomes integrated into school platforms, applications and workflows. Where AI tools connect to organisational data, systems or APIs, weaknesses in permissions, integrations or configurations can create new pathways for unauthorised access or misuse.
Third, schools face increasing data leakage risk through ‘shadow AI’, where staff or students use publicly available AI tools outside the organisation’s approved technology environment. Information entered in these services could include student records, assessment information, safeguarding information, employee data or other confidential material.
The approach therefore should combine technical security controls with AI and privacy governance: approved platforms, identity and access controls, restrictions on unauthorised AI services, vendor due diligence, monitoring and continuous awareness for both staff and students.
What cybersecurity risks do AI agents and digital personas introduce in education?
AI agents and digital personas can create valuable opportunities in education by enabling more personalised interactions, automating routine activities and improving how users engage with digital services. Their increasing capability, however, changes the cybersecurity risk profile because they can move beyond generating content to accessing information, interacting with systems and performing actions on behalf of users.
For AI agents, the key consideration is ensuring that the level of access and autonomy remains proportionate to their intended purpose. An agent connected to email, student information systems or learning platforms could have wider impact if it is over-privileged or manipulated through compromised instructions or data.
Digital personas similarly offer new ways of engaging users, but realistic AI-generated voice, video and communications also make identity and authenticity more difficult to establish.
The objective is therefore to enable these capabilities safely through appropriate permission boundaries, traceability and stronger verification for sensitive interactions.
How can schools protect AI-powered digital identities from impersonation or misuse?
Schools can support the safe adoption of AI-powered identities by creating trusted environments in which users, systems and AI agents can be appropriately authenticated and their activities remain accountable.
This starts with distinguishing human, system/service and AI-agent identities and applying controls proportionate to their roles. AI-enabled platforms can be integrated with centralised identity management, multi-factor authentication, role-based access, least privilege and controlled identity lifecycle management, while AI agents should only access the systems, data and functions required for their intended purpose.
Schools should also establish appropriate boundaries around the use of student and staff images, voice recordings and other identifiable content in AI tools, supported by awareness of deepfakes, cloned voices and fraudulent profiles.
For higher-risk interactions, independent verification remains important, alongside monitoring, escalation and takedown processes where school brands or identities are misused externally.
How should schools protect student data when using personalised AI systems?
Personalised AI systems can be valuable in education, but because they may continuously learn from or respond to information about individual students, child protection and privacy controls need to be built into their design.
Typically, schools should limit the student data that enters prompts, context windows, retrieval systems and model outputs, and ensure that identifiable student information is not retained or used for general model training unless specifically assessed and authorised. Particular attention should also be given to inferred data, such as behavioural, emotional, learning or wellbeing profiles, as these can be more sensitive than the original information provided.
For children, systems should include age-appropriate interfaces, content safeguards, clear boundaries on profiling, and appropriate controls over prompts, transcripts and generated outputs. Where AI identifies potential safeguarding concerns, harmful content or sensitive disclosures, there should be defined escalation to trained staff rather than relying on automated action.
Higher-risk uses should also be subject to child-focused privacy and security impact assessments, with reassessment where there are material changes to the model, data used, functionality or training approach.
How can education providers defend AI systems against prompt injection and manipulation?
Prompt injection should be treated as an AI application-security issue requiring multiple layers of defence. An AI system should not trust an instruction simply because it appears within the model context.
System instructions, user prompts, retrieved content and external data should be appropriately separated, with untrusted content prevented from overriding system behaviour. Education providers should also restrict AI systems to approved tools, APIs, connectors, data sources and permitted actions, rather than allowing unrestricted access across the environment.
Controls should include input and output validation, filtering of sensi tive data, trusted and permission-aware retrieval, protection against adversarial or poisoned content, rate limiting and abuse detection, and monitoring for anomalous behaviour or attempted data extraction.
AI agents should operate with tightly scoped permissions and defined tool boundaries. Most importantly, successful prompt manipulation should not automatically result in a security breach: actions such as changing student records, releasing confidential information or executing administrative functions should require additional validation or human authorisation.
What role should human oversight play in securing AI-powered learning environments?
Human oversight is fundamental in AI-powered learning environments because AI should augment professional judgement rather than replace accountability.
A meaningful ‘human-in-the-loop’ model means more than simply reviewing an output. The person responsible should understand the system’s limitations, have sufficient context to challenge its recommendations, and be able to override, stop or disregard the AI where necessary.
This is particularly important where AI influences consequential decisions involving students, such as safeguarding, disciplinary action, admissions, significant assessment outcomes or wellbeing. In these areas, AI may support analysis, identify patterns or flag concerns, but the final decision should remain with appropriately trained staff.
Human oversight should also continue throughout the AI lifecycle to identify bias, unexpected behaviour, model drift or misuse.
The key principle is that accountability must remain clearly assigned to a person or function, particularly where AI outputs can materially affect a student.
