Generative AI in education raises new ethical risks for students, schools and society

A new review warns that while generative AI tools can support personalization, access and efficiency, their growing use in education also carries risks for student agency, critical thinking, privacy, equity and the environment.

The study, The Ethical Landscape of Generative AI in Education: A Narrative Literature Review Through the Lens of Consequentialism (2022–2026), published in AI in Education, examines international literature on generative AI in education through a consequentialist ethical lens. The review focuses on whether the benefits of tools such as ChatGPT, Gemini, Copilot and Claude outweigh their wider harms, especially when those harms fall unevenly on vulnerable learners, public institutions and the natural environment.

AI's classroom benefits come with hidden ethical costs

The review links the rapid rise of genAI in education to the period following the public release of ChatGPT in late 2022, when schools, universities and policymakers began responding to a technology that could generate essays, solve problems, write code and simulate academic reasoning at speed. Early responses often focused on cheating and assessment integrity, but the paper finds that the ethical debate has since expanded into a much wider set of concerns.

GenAI is not simply another educational technology. Unlike older digital tools, it can produce complex language, provide seemingly authoritative answers and intervene directly in tasks that once required sustained student effort. The shift changes the ethical stakes because AI systems can now affect how students think, write, learn, create and make judgments, the author points out.

GenAI can offer practical gains, including personalized support, faster feedback, improved accessibility, creative scaffolding and administrative efficiency. For students with limited access to tutoring or support, AI tools may help explain difficult concepts, generate practice material or reduce barriers to participation. Teachers can use the tools for planning, drafting resources and managing workload.

However, these gains cannot be assessed in isolation. A consequentialist approach requires weighing them against costs that are often less visible, including the carbon and water footprint of AI infrastructure, algorithmic bias, misinformation, overdependence, loss of learner agency, weaker critical and creative thinking, data surveillance and the growing role of profit-driven technology companies in public education

According to the study, the environmental issue is one of the least visible yet one of the most consequential risks. Large AI models depend on energy-intensive data centers and significant water use for cooling. The review notes that these costs are rarely discussed in schools or universities even though students and educators are increasingly encouraged to use AI tools. In education, it creates an ethical contradiction: institutions may promote sustainability while relying on systems whose environmental impact remains poorly understood by users.

The review also raises concerns about bias and misinformation. GenAI systems are trained on large datasets drawn from the internet and other sources. Those datasets can contain stereotypes, cultural assumptions, ideological patterns and factual errors. When AI-generated content enters classrooms as a learning resource, students may absorb biased or inaccurate information under the impression that it is neutral or authoritative.

Is this concerning? Yes, because AI outputs often appear polished and confident. A fluent answer may mask false information, fabricated citations, narrow cultural framing or hidden assumptions. The review warns that this can weaken students' ability to distinguish between reliable knowledge and machine-generated plausibility.

Student agency and critical thinking face growing pressure

With generative AI becoming more capable, students may turn to it not only for help but for the completion of core learning tasks. The danger is not simply that students may use AI to cheat, but that they may begin to bypass the intellectual struggle through which learning takes place. If students routinely depend on AI to draft essays, summarize readings, solve problems or generate arguments, they may lose confidence in their own judgment. Over time, AI use may shift from support to dependency, especially when learners begin to treat machine-generated outputs as the default starting point for thinking.

Education is not only about producing correct answers or polished texts, it's also about developing the ability to question, evaluate, reason, revise and create. GenAI can support these goals when it is used as a tool for reflection, feedback or debate, but it can undermine them when it becomes a substitute for the student's own cognitive effort.

The paper notes that AI can enhance learning when students are required to critique its outputs, test its assumptions, compare sources and explain their own reasoning. It becomes harmful when learners accept its answers passively or use it to avoid difficult tasks. The difference depends heavily on pedagogy, assessment design and teacher guidance.

The education sector has moved beyond a simple debate over bans. The emerging question is how AI can be integrated responsibly without weakening the deeper aims of education. For this, students need to learn not only how to use AI, but how to question it.

Data privacy and surveillance add another layer of concern. When students submit prompts, drafts, questions or personal learning data into commercial AI platforms, they may be exposing sensitive information to systems that operate beyond the direct control of schools and universities. The review warns that student interactions can become part of a larger data economy in which learning behavior is collected, analyzed and potentially monetized.

Students may not fully understand how their data are used, stored or shared. Institutions may adopt tools before developing clear governance systems. Teachers may be asked to integrate AI into learning without adequate guidance on privacy, consent or accountability.

Another concern is that major AI platforms are largely developed by private companies with business models shaped by scale, market capture and user data. Their growing presence in education raises questions about who controls learning infrastructure, who benefits financially and whether educational values are being reshaped by corporate priorities.

The author states that education systems should resist the uncritical commodification of teaching and learning. If AI adoption is driven mainly by market pressure or institutional fear of falling behind, schools and universities risk handing core educational functions to commercial platforms without enough public oversight.

Why it matters for AI policy in education

Education systems need stronger ethical governance before generative AI becomes deeply integrated in everyday learning. The issue is no longer whether students and teachers will use AI, it's about whether it will be used in ways that serve learning, equity and public good, or in ways that produce short-term efficiency while creating long-term harm.

The author proposes that institutions assess AI adoption by weighing benefits against harms across several dimensions, including learning outcomes, student autonomy, privacy, bias, environmental sustainability and equity. This means AI policies should go beyond academic integrity rules. They should address procurement, data governance, environmental impact, teacher training, student rights and accountability.

Students should learn how generative AI systems work, why they can produce errors, how bias enters outputs, what data risks exist and what environmental costs are linked to large-scale AI use. This literacy should not be limited to computer science or higher education. It should become part of general education because AI is already shaping how students access and produce knowledge.

Responsible AI use depends heavily on structured pedagogical design. Educators need training to create assignments that require critical engagement with AI rather than simple outsourcing. Assessments may need to place greater emphasis on process, reflection, oral defense, in-class reasoning, source evaluation and original judgment.

Institutions also need stronger privacy protections. Schools and universities should scrutinize AI platforms before adoption, ensure that student data are not exploited without meaningful consent and prefer tools that provide transparency over data use. Where possible, public institutions may need to consider open-source, locally hosted or education-specific AI systems that reduce dependence on commercial platforms.

Environmental accountability should also become part of AI governance. Education systems should not ignore the energy and water demands of AI infrastructure. Institutions can ask providers for clearer sustainability disclosures, include environmental criteria in procurement and teach students about the resource costs behind digital convenience.

Equity is another major concern. Generative AI may widen gaps if wealthier institutions and students gain better tools while under-resourced learners rely on lower-quality or poorly governed systems. At the same time, vulnerable groups may bear greater risks from biased outputs, surveillance and reduced human support. Policies must ensure that AI adoption does not deepen existing inequalities.

A tool should not be judged only by whether it improves speed, convenience or short-term performance. It should be judged by whether it strengthens or weakens the deeper purposes of education: independent judgment, critical inquiry, creativity, ethical awareness and democratic participation.