Why Fixing AI Bias Requires More Than Better Data, Fairness Scores and Human Oversight

An artificial intelligence (AI) system can deliver statistically accurate predictions, satisfy established fairness criteria and still reproduce discrimination through the very information it was designed to process. The problem may emerge long before an algorithm evaluates a job applicant, assesses a patient's medical needs or generates responses in different languages. It begins with decisions about which characteristics deserve measurement, which categories represent people and which outcomes technology should prioritise.

A research paper titled "When Algorithms Cannot Be Fair: Constitutive and Contingent Discrimination and the Limits of Less-Discriminatory Alternatives," published in the journal AI & Society by Alfredo Di Giorgio of the University of Salento, Italy, challenges the foundations of conventional algorithmic fairness assessments. The study argues that concentrating on prediction accuracy, statistical disparities and corrective adjustments overlooks a more fundamental question: whether the conceptual architecture of an AI system already contains the conditions for discriminatory outcomes.

Drawing on cases involving healthcare algorithms, automated recruitment, image recognition and multilingual language models, the research develops a framework for distinguishing discrimination that can be corrected through alternative designs from disparities that remain unavoidable under specified technological and institutional constraints. Its central contribution is a proposed shift in AI accountability from testing the fairness of existing systems to examining whether less-discriminatory systems could have been designed in the first place.

The discrimination hidden inside an algorithm's definition of reality

Before an algorithm can evaluate human behaviour or social conditions, developers must translate complicated realities into categories, variables and numerical representations. Concepts such as merit, medical need, gender and risk carry contextual meanings that cannot necessarily be captured through predefined computational measures. Yet automated systems depend on precisely these simplifications to operate consistently across large populations.

The author identifies a fundamental limitation in this process. Once developers establish the categories a system recognises, information falling outside those categories becomes difficult or impossible for the system to interpret. A recruitment algorithm may recognise qualifications and employment history without capturing the circumstances under which those achievements occurred. A healthcare model may measure expenditure without adequately representing illness among patients facing barriers to treatment.

The study describes this limitation through the concept of computational fog, where deficiencies in a system's underlying categories remain invisible within the analytical framework built upon them. Information excluded during design cannot simply be recovered through improved prediction techniques. Apparent anomalies may therefore be treated as statistical noise even when they reveal shortcomings in how the underlying problem was represented.

This challenges the assumption that greater transparency or better training data necessarily produces fairer AI. Even a fully interpretable algorithm can rely on inadequate categories, while human oversight may struggle to correct decisions when crucial contextual information has already been discarded. According to the research, computational systems make these predefined interpretations consequential by applying them uniformly and at scale, without the case-by-case reconsideration available to human decision-makers.

When existing designs reach their limits

The study's most important theoretical contribution is a distinction between contingent and constitutive algorithmic discrimination. Contingent discrimination occurs when another admissible design can fulfil the same legitimate objective while reducing an identified disparity. Constitutive discrimination arises when no such alternative is available within a specified range of designs, given the available evidence and constraints.

These categories are not permanent labels attached to entire technologies. Their meaning depends on the system's legitimate purpose, the particular disparity being assessed and the alternative designs considered technically, legally and institutionally feasible. Technological improvements or changes in those constraints can transform a disparity previously regarded as unavoidable into one that can be reduced.

Evidence from healthcare illustrates the practical importance of this distinction. Di Giorgio examines earlier research into a commercial algorithm used to identify patients requiring additional care management. The system predicted future healthcare expenditure as a substitute for medical need, despite unequal healthcare access causing similarly ill patients to generate different levels of spending.

The earlier investigation found that correcting this bias would increase the proportion of Black patients receiving additional assistance from 17.7% to 46.5%. When researchers changed the prediction target to incorporate measures of health rather than expenditure alone, the disparity fell by 84%. The original model could predict spending effectively while failing to identify need equitably because the underlying measurement choice was flawed.

Automated recruitment reveals a related difficulty. The paper revisits Amazon's abandoned experimental hiring system, which penalised indicators associated with women because its assessment criteria reflected historically male-dominated recruitment patterns. Removing explicit gender-related indicators did not prevent the system from learning subtler proxies. The example demonstrates how discrimination can survive individual technical corrections when the underlying representation of professional merit remains tied to historical selection practices.

Language models expose a different combination of avoidable and persistent inequalities. Research examined by Di Giorgio found that equivalent content could require up to 15 times more tokens in certain languages, creating disadvantages where access and computational resources are measured by token counts. Better-balanced training and vocabulary allocation can reduce some disparities, while residual differences remain unresolved under current shared-vocabulary architectures. The study therefore cautions against treating every disadvantage as either entirely fixable or inherently permanent.

Why AI regulation must examine decisions made before the model exists

Existing accountability mechanisms generally assess AI systems after their principal design assumptions have been established. They evaluate training data, model behaviour, discriminatory outputs and human oversight arrangements. Although these requirements can identify significant problems, Di Giorgio argues that they do not necessarily interrogate whether the categories and objectives selected during development were appropriate.

The European Union's Artificial Intelligence Act provides the main regulatory context for the study. Its risk-based framework establishes requirements relating to risk management, data governance, transparency and human oversight for designated high-risk applications, including areas such as employment and credit scoring. The paper identifies opportunities to strengthen these arrangements by examining representational decisions before data collection and system architecture become finalised.

Di Giorgio proposes ontological auditing, an upstream evaluation of the assumptions through which an algorithm represents its operating environment. Its central instrument, an Ontological Impact Statement, would document the phenomena represented by selected variables, the information excluded, possible relationships with protected characteristics and the alternative representations considered during development.

The proposal introduces an additional accountability requirement: developers claiming that a discriminatory outcome could not reasonably have been avoided would need to demonstrate which alternatives were examined, why they were rejected and what evidence supports the conclusion. An assertion that no better design exists would no longer be sufficient without a documented examination of available options.

Such a requirement could change how organisations justify socially consequential AI systems. Rather than treating fairness solely as a measurable property of completed models, institutional reviews would also scrutinise the decisions that define the models' objectives. The research suggests integrating this scrutiny into existing EU AI Act provisions governing risk management and data quality, with interdisciplinary review involving technical specialists, domain experts, legal scholars and representatives of affected communities.

The proposed approach also carries practical difficulties. Determining whether an alternative design is genuinely available requires more than identifying a hypothetical technical solution. Developers must consider whether alternatives fulfil the intended purpose and whether relevant constraints are justified. The study warns that defining the available design space too narrowly could make discriminatory outcomes appear unavoidable simply because potentially better approaches were excluded from consideration.

Beyond technical fixes: The unfinished agenda for algorithmic fairness

The research challenges a longstanding tension in algorithmic governance: mathematical fairness standards can conflict even when systems are functioning as designed. Established impossibility results demonstrate that certain statistical fairness conditions cannot all be satisfied simultaneously when underlying group rates differ and prediction is imperfect. Di Giorgio does not dispute those mathematical findings but questions whether their assumptions should automatically determine the boundaries of possible fairness.

Population categories, prediction targets and measured outcomes are themselves products of prior choices. Changing those choices can alter the conditions under which fairness is evaluated, although no such change guarantees that all disparities can be removed. The research therefore separates mathematical constraints within a fixed system from broader questions about whether a different representation of the underlying problem would be legitimate and feasible.

This argument has particular relevance for systems operating across different social and linguistic environments. The language-model evidence demonstrates how design conventions can translate into unequal computational costs, while the clinical example reveals how institutional inequalities can distort apparently objective measurements. Both cases indicate that technical performance must be interpreted alongside the social conditions shaping the information algorithms receive.

Nevertheless, the paper is a conceptual investigation rather than a new empirical evaluation of deployed systems. Its examples draw on existing research, and the proposed auditing framework has not yet undergone institutional testing. The administrative costs, practical feasibility and effectiveness of Ontological Impact Statements remain unresolved. The study also acknowledges that disputes over legitimate objectives and acceptable measures of discrimination cannot be settled through technical analysis alone.

Further research would need to test the framework in actual development processes, establish workable standards for evaluating alternative representations and examine whether upstream scrutiny produces improvements beyond conventional fairness auditing. Particular attention would be required when commercial constraints are treated as technological necessities, since this distinction can determine whether a discriminatory outcome is regarded as unavoidable.