AI failures are predictable - and preventable

One of the core findings of the study is that many AI failures could be predicted early through a systematic evaluation of key performance factors. The researchers propose that AI teams consider two main dimensions: expected model performance and the minimum required performance needed to avoid harm.

Artificial Intelligence (AI) is advancing at an unprecedented pace, promising groundbreaking efficiencies across industries. However, a critical challenge remains: many AI systems fail to meet expectations, leading to unintended consequences and missed opportunities.

A recent study titled "AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development", published in CHI Conference on Human Factors in Computing Systems," sheds light on these failures. Conducted by researchers Devansh Saxena (University of Wisconsin-Madison), Ji-Youn Jung, Jodi Forlizzi, Kenneth Holstein, and John Zimmerman (Carnegie Mellon University), the study explores AI mismatches - the gap between a system's expected and actual performance. Through an analysis of 774 AI cases, the researchers propose a framework to mitigate risks early in the AI development process.

AI mismatches: A systemic challenge

AI mismatches occur when an AI system's real-world performance does not align with its intended purpose, leading to inefficiencies, biases, or even harmful outcomes. The study identifies multiple risk factors contributing to these mismatches, including poor data quality, unaccounted variables, and unrealistic performance expectations. Many AI failures stem from models being deployed in contexts where they cannot deliver meaningful benefits due to limitations in the training data or fundamental gaps in understanding the problem at hand. The researchers argue that AI harm is often preventable if these mismatches are identified during the conceptualization stage rather than after deployment.

To address this issue, the study introduces a structured framework for evaluating AI concepts before development begins. The researchers developed seven matrices that map the relationships between key risk factors and high-risk areas. These matrices help AI teams anticipate potential failure points, assess feasibility, and refine AI projects before investing significant resources.

Identifying high-risk AI concepts before development

One of the core findings of the study is that many AI failures could be predicted early through a systematic evaluation of key performance factors. The researchers propose that AI teams consider two main dimensions: expected model performance and the minimum required performance needed to avoid harm. In many cases, AI concepts are pursued without a clear understanding of the quality and completeness of data required to achieve success. For example, AI-driven decision-making in high-risk domains such as healthcare, criminal justice, and social services often lacks essential human contextual inputs, resulting in poor predictions and biased outcomes.

The study highlights several real-world examples where AI mismatches led to failure. One notable case is an AI system designed for child welfare risk assessments, which misclassified families due to biased training data, leading to inappropriate interventions. Similarly, AI hiring tools have been shown to replicate existing biases in recruitment, disproportionately disadvantaging certain demographic groups. These failures underscore the need for a pre-development risk assessment framework that evaluates whether AI can realistically achieve the intended benefits.

Role of human-centered AI evaluation

A key recommendation from the study is the adoption of a human-centered approach in AI evaluation. Instead of relying solely on traditional machine learning metrics like accuracy or precision, AI teams should assess how well a model aligns with human needs and decision-making processes. The researchers suggest that AI systems should be evaluated based on:

• The expected frequency and severity of errors
• The system's ability to mitigate or detect errors effectively
• The impact of unobserved variables on decision-making
• The cost-benefit balance between automation and human oversight

By integrating these human-centered evaluation criteria, AI teams can better anticipate risks and design systems that align with real-world complexities. The study also emphasizes the importance of transparency and interdisciplinary collaboration in AI development, ensuring that diverse perspectives contribute to identifying potential risks before deployment.

Moving towards responsible AI development

The findings of this study provide a critical framework for AI developers, policymakers, and researchers seeking to build more ethical and effective AI systems. Rather than focusing on fixing AI harms after they occur, the proposed approach shifts the focus toward preemptive risk assessment. This paradigm shift has significant implications for AI governance, regulation, and corporate responsibility, encouraging organizations to adopt a more thoughtful and measured approach to AI deployment.

Ultimately, the study underscores the need for AI developers to move beyond the hype and prioritize careful planning and evaluation. By identifying AI mismatches early, teams can prevent costly failures and ensure that AI serves as a tool for equitable and meaningful progress.