AI stereotypes are widespread and hard to eliminate

A new study sheds light on one of the most persistent and controversial challenges in artificial intelligence (AI): the presence of stereotypes in large language models (LLMs). The research moves beyond surface-level observations of biased outputs and instead examines where and how such biases are embedded within the internal architecture of widely used AI systems, revealing that stereotypes are not isolated glitches but deeply rooted features of how these models process language.

The study, titled "Can We Locate and Prevent Stereotypes in LLMs?" published on arXiv, investigates the internal workings of GPT-2 Small and Llama 3.2 models to identify the neural components responsible for generating stereotypical outputs.

Bias is not isolated but distributed across neural networks

The study challenges a common assumption in AI safety research: that bias can be traced to specific faulty components within a model. Instead, the findings show that while certain neurons and attention heads exhibit stronger responses to stereotypical content, these signals are not confined to isolated units.

Through the first experiment, the research identifies "contrastive neurons" that activate significantly more for stereotypical inputs than for anti-stereotypical or unrelated alternatives. These neurons appear across multiple layers and components of the model, including embeddings, attention heads, and feedforward networks. In some cases, activation ratios reach extreme levels, suggesting that certain parts of the network are highly sensitive to biased content.

However, these high-activation neurons do not act independently. Instead, they form part of a broader, distributed system of representation. The data shows that bias-related signals are spread across layers, with early attention layers and mid-level feedforward networks both playing key roles in encoding stereotypical associations.

The distribution of these signals varies by bias category. Race and profession-related stereotypes dominate the highest-ranking neurons, reflecting both the structure of training data and the frequency of such associations in language corpora. Gender and religion biases, while present, appear less frequently among top-ranked neurons but still exhibit strong activation patterns when detected.

This layered and distributed encoding challenges efforts to localize bias within a single part of the model. Instead of being confined to a few "bad neurons," stereotypes appear as emergent properties of the network's overall structure, shaped by patterns learned during training.

Targeted interventions reduce bias signals but fail to eliminate them

To test whether these identified neurons are responsible for biased outputs, the study conducts a series of ablation experiments, systematically removing high-impact neurons and measuring the effect on model behavior. The results reveal a surprising disconnect between detection and intervention.

Despite identifying neurons with extremely high activation ratios, removing them produces only marginal reductions in stereotypical output. In many cases, the change in bias-related behavior is less than one percent, indicating that no single neuron or small group of neurons is solely responsible for generating biased responses.

Even more striking is the finding that removing certain neurons can sometimes increase bias rather than reduce it. This counterintuitive outcome highlights the complexity of neural networks, where components interact in nonlinear ways and removing one pathway may amplify another.

The second experiment uses a probing approach to identify attention heads and neuron subsets that contribute most strongly to distinguishing between stereotypical and non-stereotypical inputs. These probes achieve high classification accuracy, reaching approximately 73 percent for GPT-2 and 80 percent for Llama models.

Further analysis reveals that a relatively small subset of attention heads, roughly 15 to 30 percent, accounts for most of the bias-related signal. Within these heads, an even smaller subset of neurons plays a critical role in classification tasks. When these neurons are removed, the model's ability to distinguish between biased and unbiased inputs drops sharply.

However, this reduction in classification accuracy does not translate into a meaningful decrease in biased text generation. While some improvement is observed in bias metrics, the overall effect remains limited, and the model continues to produce stereotypical outputs at similar rates.

This gap between detection and mitigation leads to what the study describes as an ablation paradox. Even when the components most strongly associated with bias are removed, the model retains its ability to generate biased content, suggesting that the underlying representation of bias is more resilient than previously thought.

Bias originates early and persists through model pathways

Bias is present from the very beginning of the model's processing pipeline. By analyzing embedding layers, the researchers show that stereotypical signals are already encoded in the initial token representations before any deeper processing occurs.

Classification experiments using only embedding-level data achieve accuracy levels above 70 percent, indicating that bias is embedded in the foundational representations learned during training. Positional embeddings, by contrast, show near-random performance, confirming that the bias signal is tied to semantic content rather than structural features.

This early encoding has important implications for mitigation strategies. If bias is present at the input representation stage, later interventions in the network may be insufficient to fully remove it. Instead, bias propagates through the model via multiple pathways, reinforced at each layer.

The study introduces the concept of a "bias direction" within the model's residual stream, a high-dimensional space where information is continuously updated and refined. Rather than being tied to specific neurons, bias exists as a direction within this space, supported by many different components.

Because of this redundancy, removing a subset of neurons only partially weakens the bias signal. The remaining components continue to carry and reconstruct the same information, allowing the model to maintain its behavior even after targeted interventions.

Residual connections further complicate the picture. These connections allow information to bypass individual layers, ensuring that bias signals encoded early in the process can persist throughout the network. Even if one pathway is disrupted, alternative routes can preserve the same underlying representation.

This architecture makes bias highly resistant to localized fixes, reinforcing the idea that effective mitigation will require more comprehensive approaches.

Implications for AI safety and future research

Current approaches to bias mitigation often focus on identifying and removing problematic components within a model. This study suggests that such strategies may be insufficient, as bias is not confined to discrete units but embedded across the network's entire structure.

The research also highlights the role of training data in shaping model behavior. Since bias is present in initial embeddings, it likely originates from patterns in the data used to train the model. Addressing bias at this stage may be more effective than attempting to remove it after the model has been trained.

The study simultaneously points to new directions for intervention. One promising approach involves transforming the representation space itself, rather than targeting individual neurons. Techniques such as sparse autoencoders could help disentangle overlapping features and isolate bias-related components more effectively.

Another avenue involves developing real-time detection systems that monitor model outputs during inference. By identifying biased responses as they occur, these systems could provide a layer of oversight without requiring fundamental changes to the model architecture.

The findings also highlight wider concerns around accountability and transparency in AI systems. As language models see broader adoption, it becomes increasingly important to understand the reasons behind their outputs to ensure they are deployed responsibly.