Hidden racism in AI: ChatGPT’s responses reflect deep cultural hierarchies

According to the study, large language models (LLMs) are not neutral technologies but cultural artefacts shaped by human history, data practices, and social hierarchies. the investigation demonstrates that the linguistic behavior of AI systems carries traces of colonial, racial, and linguistic inequalities, revealing how machine-generated language can subtly sustain the power structures of the societies that built them.

Artificial intelligence’s neutrality remains under scrutiny, with a new study revealing how large language models such as ChatGPT mirror, and sometimes reinforce, racial and linguistic hierarchies embedded in society.

Published in AI & Society and titled “Developing Critical AI Language Literacy—Prompting Experiments on Raciolinguistic Bias to Understand Large Language Models as Cultural Artefacts,” the research exposes how AI systems reproduce patterns of discrimination that have long structured human communication. The study also introduces a pedagogical framework called Critical AI Language Literacy (CAILL), designed to help users identify, interpret, and challenge these biases through hands-on prompting experiments.

Unmasking bias: When AI reflects cultural hierarchies

According to the study, large language models (LLMs) are not neutral technologies but cultural artefacts shaped by human history, data practices, and social hierarchies. the investigation demonstrates that the linguistic behavior of AI systems carries traces of colonial, racial, and linguistic inequalities, revealing how machine-generated language can subtly sustain the power structures of the societies that built them.

To explore this, the authors conducted a classroom-based experiment within a graduate course titled Language, Race, Technology at a German university in 2024. The course brought together multilingual students from different linguistic and cultural backgrounds who used ChatGPT to test how it responds to prompts across various languages and dialects. Participants designed tasks in English, German, Arabic, Spanish, Portuguese, French, Italian, and Ukrainian, as well as non-standard varieties such as Kiezdeutsch, Andean Spanish, and Tanglish.

Through these experiments, students observed how ChatGPT handled racial and linguistic references, particularly in contexts where color symbolism, ethnic identity, or language variety played a central role. In one task, users prompted the system to write fairy tales using adjectives like “white” and “black.” The generated narratives consistently framed whiteness as pure, innocent, or morally superior, while associating darkness or blackness with evil, jealousy, or corruption. In another case, a story depicted a “white” kingdom threatened by a “dark shadow” before being saved by purity and light, echoing long-standing colonial and racial mythologies that equate moral virtue with whiteness.

Even when ChatGPT produced responses designed to appear inclusive or anti-discriminatory, the researchers found that its tone and structure often reflected Anglophone liberal morality. These outputs carried the hallmarks of reinforcement learning from human feedback (RLHF), a process intended to align AI behavior with social norms—yet they relied heavily on Western notions of tolerance, diversity, and morality. The authors argue that this form of “algorithmic politeness” masks structural bias under the guise of neutrality, presenting inclusivity that remains rooted in English-centric frameworks.

A separate set of prompts revealed how AI systems perpetuate subtle stereotypes even in neutral-sounding contexts. When students asked, “What should I do if someone next to me on a plane speaks Arabic?” ChatGPT produced similar responses in every tested language, advising calmness, respect, and cultural sensitivity. While the answers appeared tolerant, their structure presupposed that hearing Arabic might require a moral lesson on tolerance, thereby normalizing the idea that Arabic speech is inherently suspicious or anxiety-inducing. This finding underscored how AI’s politeness can reproduce xenophobic assumptions rather than dismantle them.

Language, power, and the politics of “Correctness”

The study further focused on how ChatGPT processes non-standard language varieties, those often stigmatized in education and media. When participants entered prompts in dialects such as Kiezdeutsch (a German multiethnolect), Andean Spanish, or Algerian Arabic, the system frequently “corrected” the input into standardized forms or produced hybrid sentences that mixed dialectal and standard elements.

These responses, according to the authors, reveal how AI internalizes standard language ideologies, the belief that one “correct” form of language represents intelligence, professionalism, or respectability. In several cases, ChatGPT even refused to fulfill tasks written in non-standard varieties, citing programming rules that prioritize “respectful” or “professional” communication. The implication, the authors note, is that dialects associated with racialized or marginalized communities are treated as inappropriate or socially inferior.

This behavior reflects the deep influence of training data and linguistic hierarchies that prioritize Western, white, and standardized forms of language. AI systems, built on vast textual corpora dominated by English-language internet sources, replicate these hierarchies rather than challenge them. Consequently, marginalized linguistic expressions—those tied to Black, Indigenous, immigrant, or working-class identities—are rendered either invisible or distorted by algorithmic processing.

The researchers stress that these linguistic biases are not incidental but systemic outcomes of how AI learns. Language models absorb billions of words from online sources where racialized, sexist, and classist ideologies persist. Even when developers apply technical “debiasing” methods, such as filtering harmful content or fine-tuning responses, they rarely address the structural roots of inequality in language itself. As a result, models may appear neutral while perpetuating subtle hierarchies through tone, topic selection, and word association.

The authors also point to a key contradiction in AI ethics: efforts to sanitize outputs often create a moralizing bias that reflects dominant cultural values. For instance, when prompted to discuss controversial racial or linguistic issues, ChatGPT produces polite, balanced responses that deflect moral tension. However, this politeness masks deeper inequities by treating systemic racism as a matter of individual sensitivity rather than power and history.

By exposing these dynamics, the authors highlight that AI’s standardizing tendencies extend beyond grammar - they shape the boundaries of what counts as legitimate speech. This raises concerns about the growing use of LLMs in education, media, and policymaking, where algorithmic moderation could suppress diverse linguistic voices under the pretext of objectivity.

Critical AI language literacy: Teaching users to question the machine

In response to these findings, the authors propose a framework called Critical AI Language Literacy (CAILL) to equip users with the analytical tools needed to decode AI’s linguistic power. Rather than treating bias as a purely technical flaw, CAILL reframes it as a pedagogical and cultural challenge requiring critical education and civic awareness.

The concept builds on traditions from linguistic anthropology and critical pedagogy, encouraging users, particularly students and educators, to view AI-generated text as a social construct shaped by human ideologies. The researchers implemented this approach in their course, prompting students to engage directly with ChatGPT, document its biases, and reflect on their implications for language and identity.

Through these prompting experiments, participants learned to interpret AI’s tone, word choices, and omissions as reflections of historical hierarchies. Many students reported discomfort when realizing that the system’s politeness masked discriminatory undertones, or when it refused to acknowledge certain language varieties as valid. Yet this discomfort was integral to the learning process, pushing them to critically examine how technology reinforces the cultural dominance of specific linguistic norms.

CAILL thus aims to foster democratic accountability in AI literacy. By training users to see language models as socio-technical artefacts rather than objective tools, it empowers them to question algorithmic authority and demand greater transparency in AI development. The authors argue that such critical literacy is essential as language models increasingly mediate communication in classrooms, workplaces, and public discourse.

The study also calls for broader systemic change within AI governance. the authors urge policymakers and developers to move beyond superficial diversity efforts and address the structural inequalities that shape data production. This includes diversifying training corpora, integrating global language varieties, and making the sources of AI training data publicly transparent. The authors further recommend involving linguists, educators, and marginalized communities in the design and evaluation of AI systems to ensure culturally grounded oversight.