Hidden barriers to AI success in healthcare and social care

Data inconsistencies arise when formats, units, or collection practices change over time, undermining model reliability. Poor environmental conditions, such as inadequate lighting or constrained workspaces, can degrade data quality. Data restrictions imposed by ethical, legal, or organizational rules further limit what can be modeled, forcing trade-offs that reduce predictive usefulness.

Machine learning has been promoted as a transformative force for healthcare and social welfare systems, promising earlier diagnoses, better targeting of services, and more efficient use of limited public resources. Despite a decade of experimentation, large-scale adoption remains uneven, and many deployed systems fail to achieve their intended impact once they reach frontline practice.

A new academic study, published as an arXiv preprint, states that the problem is not simply flawed algorithms or lack of technical maturity, but a deeper mismatch between machine learning systems and the realities of care work. The study, titled “Sociotechnical Challenges of Machine Learning in Healthcare and Social Welfare,” presents a structured framework to explain why machine learning tools so often struggle in real-world healthcare and social welfare settings. 

Why technical performance alone does not guarantee impact

Many machine learning tools deployed in healthcare perform well in controlled testing environments but encounter persistent friction once embedded in professional workflows. Existing explanations for this gap tend to fall into two camps. One attributes failure to designers’ limited understanding of social contexts, while the other points to fundamental limits of technology when applied to complex human systems. The authors argue that both explanations are incomplete because they focus too narrowly on design-stage decisions and overlook how systems evolve through use.

Instead, the study defines sociotechnical challenges as problems that emerge when machine learning tools are repeatedly used within care practices. These challenges are not isolated technical bugs or ethical edge cases. They arise from ongoing interactions between tools, professionals, organizational norms, and material conditions. In this view, machine learning systems do not simply enter a static environment; they reshape and are reshaped by the practices they are meant to support.

To capture this dynamic, the authors introduce the concept of an ML-enabled care pathway. This pathway represents the sequence of activities involved when practitioners gather data, interpret algorithmic outputs, and initiate actions such as interventions or monitoring. Sociotechnical challenges, the study shows, appear at different points along this pathway, often overlapping and reinforcing one another.

During data collection, challenges emerge when real-world conditions fail to match designers’ assumptions. Data inconsistencies arise when formats, units, or collection practices change over time, undermining model reliability. Poor environmental conditions, such as inadequate lighting or constrained workspaces, can degrade data quality. Data restrictions imposed by ethical, legal, or organizational rules further limit what can be modeled, forcing trade-offs that reduce predictive usefulness.

When practitioners interpret model outputs, additional tensions surface. Clinicians and social workers often rely on contextual cues that are invisible to algorithms, leading to hidden predictors that shape decisions outside the system’s awareness. Mismatched objectives occur when the outcomes optimized by a model differ from what practitioners prioritize in their daily work. Decision point disconnects arise when algorithmic outputs fail to reach the right person at the right moment, rendering technically sound tools operationally irrelevant.

At the action stage, problems intensify. Practitioners may attempt to draw causal conclusions from systems designed only for prediction, resulting in insufficient causal information for effective intervention. Value conflicts surface when machine learning tools expose disagreements about professional autonomy, responsibility, and acceptable risk. Across all stages, broader issues such as disrupted workflows, limited understanding of system behavior, and increased administrative labor compound resistance and reduce trust.

A framework built from practice, not theory alone

The authors' framework is based on four interconnected research phases. These include in-depth fieldwork within a social welfare organization deploying a machine learning tool, a qualitative review of 22 longitudinal deployment studies, and co-design workshops with healthcare and social welfare practitioners.

The fieldwork component involved months of participant observation and interviews with data scientists, project managers, consultants, and frontline staff. This allowed the researchers to observe sociotechnical challenges as they unfolded, rather than relying on retrospective accounts. The literature review extended these insights across healthcare and social welfare contexts, identifying recurring patterns in how machine learning tools are adopted, adapted, or abandoned. The co-design workshops then tested and refined the framework by asking practitioners to classify challenges based on when they arise during interaction with machine learning systems.

The result is a categorization of 11 sociotechnical challenges, organized along the ML-enabled care pathway. However, the study goes further by offering a process-oriented explanation of how these challenges emerge. It identifies three interacting processes that shape outcomes: determining design, grappling with constraints, and deviating from scripts.

Determining design refers to the social and political negotiations that translate organizational priorities into system features. Early design choices often reflect managerial goals, such as efficiency or cost reduction, rather than the nuanced needs of frontline decision-making. Grappling with constraints highlights the tension between rigid technical representations and the fluid nature of care work. Even well-intentioned designs run into limits when complex human phenomena must be reduced to quantitative inputs. Deviating from scripts captures how practitioners adapt tools through improvisation, workarounds, and selective use, often diverging from designers’ expectations in order to make systems usable in practice.

These processes explain why sociotechnical challenges persist even when systems are technically sound and ethically reviewed. Problems do not originate solely from ignorance or malice; they emerge through routine use as people and technologies interact within institutional settings.

Implications for the future of AI in care systems

The study challenges the assumption that improving model accuracy or explainability alone will resolve adoption barriers. While technical improvements matter, they do not address deeper mismatches between machine learning systems and professional practice. Without attention to workflows, norms, and organizational incentives, even highly accurate tools may fail to gain traction.

Next up, the framework suggests that sociotechnical challenges should be anticipated rather than treated as unexpected failures. By recognizing common patterns across deployments, organizations can design evaluation and governance processes that monitor how systems are used over time, not just how they perform at launch. This includes tracking additional labor demands, shifts in decision-making authority, and changes in professional judgment.

Third, the study reframes the role of practitioners in AI deployment. Frontline staff are not passive users who either accept or reject technology. They actively shape system outcomes through interpretation, adaptation, and resistance. Successful machine learning deployments, the authors argue, must engage practitioners as co-constructors of practice rather than as endpoints of technical delivery.

The research also speaks to ongoing debates about accountability and trust in AI-assisted decision-making. By showing how challenges arise from misalignment between tools and practice, the study highlights the risks of over-reliance on algorithmic outputs without understanding their limits. It also cautions against simplistic calls for human override mechanisms that ignore organizational realities such as time pressure and workload.