AI’s next test in healthcare is trust, accountability and real-world safety

AI is moving deeper into healthcare administration and clinical informatics, but its ability to improve patient care will depend on stronger governance, safer data systems and continuous human oversight, according to a new review published in Healthcare by Hanadi Aldosari of Taibah University.

The study, titled Artificial Intelligence in Healthcare Administration and Clinical Informatics: A Critical Review and Governance Roadmap, examines AI use across disease diagnosis, clinical decision support, treatment personalization, drug discovery, hospital operations, remote monitoring, public health surveillance and mental health tools, while warning that real-world adoption remains constrained by fragmented data, algorithmic bias, privacy risks, weak interoperability and unclear accountability.

AI tools are expanding from diagnosis to hospital operations

The review identifies diagnostic medicine as one of the strongest areas of AI progress, particularly in image-rich specialties such as radiology, ophthalmology, dermatology, pathology and cardiology. Deep learning systems have shown strong performance in selected settings, including lung cancer screening, breast cancer screening, diabetic retinopathy detection, skin lesion classification and arrhythmia detection.

In ophthalmology, autonomous AI-based diabetic retinopathy screening is highlighted as a milestone because it moved from research toward regulated clinical use. In radiology and pathology, AI systems can help detect abnormalities in scans and tissue images, reduce variability and support high-volume screening workflows. In cardiology, AI-enabled electrocardiogram analysis can assist with arrhythmia detection and early identification of hidden cardiac dysfunction.

The paper cautions, however, that strong results in controlled datasets do not guarantee safe performance in hospitals. Models can behave differently across institutions because of changes in patient populations, imaging equipment, clinical protocols, coding practices and disease prevalence. This makes external validation, local testing and post-deployment monitoring essential before systems are scaled.

Apart from diagnosis, AI is increasingly being used in clinical decision support and treatment personalization. Machine learning models can combine clinical history, laboratory values, imaging findings, genomic data and treatment response patterns to support risk stratification and individualized care pathways. In oncology, AI can help link tumor genomics, digital pathology, imaging features and clinical records to guide therapy decisions. In diabetes care, AI can support continuous glucose monitoring, predict hypoglycemia or hyperglycemia and assist closed-loop insulin delivery systems.

Drug discovery is another major area of application. The review points to AI's role in virtual screening, molecular design, toxicity prediction, protein structure prediction and drug repurposing. Systems such as AlphaFold have changed biomedical research by expanding access to predicted protein structures, while deep learning models have helped identify potential antimicrobial compounds and new therapeutic candidates.

Healthcare administration is also emerging as a critical AI use case. Predictive analytics can help forecast admissions, estimate length of stay, identify discharge readiness, predict readmission risk and support bed management. These tools could help hospitals reduce emergency department congestion, allocate staff more effectively and improve resource planning. AI can also support scheduling, supply classification, inventory management and operational dashboards.

Natural language processing is becoming important in electronic health record management. Much of the clinical value in EHR systems remains locked in unstructured notes, discharge summaries, referral letters, pathology reports and radiology reports. NLP tools can extract relevant clinical information, summarize patient histories and reduce documentation burden. The review warns that these systems must remain human-supervised because inaccurate summaries, missing context or generated errors can create clinical and legal risks.

Remote monitoring and telemedicine are also expanding AI's role outside hospitals. Wearables, mobile platforms and connected sensors can track heart rate, oxygen saturation, activity, sleep and cardiac rhythm. AI can analyze those data streams to detect deterioration, flag possible atrial fibrillation, support chronic disease management and help clinicians prioritize urgent cases. But device variability, false alerts and unequal access to digital tools remain major concerns.

The review also covers public health and mental health. AI can support outbreak detection, pandemic preparedness, vaccination planning and population-level surveillance. In mental health, conversational agents and symptom-monitoring tools may improve access to low-threshold support, but the paper stresses that they cannot replace professional care and require safeguards for crisis management, privacy and clinical safety.

Bias, privacy and weak infrastructure remain major barriers

The review warns that AI adoption in healthcare is being slowed by technical, ethical and organizational barriers. Poor data quality is one of the biggest problems. Healthcare data are often fragmented across hospitals, laboratories, imaging systems, registries, billing platforms and EHR vendors. These systems may use different coding practices, terminology, formats and documentation standards, making it difficult to build reliable AI tools.

Interoperability is therefore crucial to responsible AI deployment. Standards such as Fast Healthcare Interoperability Resources can help structure health data exchange, but the review stresses that standards alone are not enough. Healthcare organizations also need consistent data definitions, transparent data provenance, clear data access rules and strong governance for secondary use of clinical information.

Algorithmic bias is another major risk. AI models trained on unrepresentative data can reproduce or amplify existing health inequalities. A model may perform well on average while failing specific groups because of race, age, sex, skin tone, income, geography or disease pattern differences. Dermatology models trained mainly on lighter skin tones, for example, may be less reliable for darker skin tones. Tools developed in high-income health systems may also fail in lower-resource settings if they are not locally validated.

The review emphasizes that fairness cannot be treated as an optional add-on. AI systems must undergo subgroup analysis, fairness audits, external validation and continuous equity monitoring. Dataset transparency is needed so developers, clinicians, regulators and patients can understand who is represented in training data and where the model may be weak.

Explainability remains another unresolved challenge. Many AI systems, especially deep learning and generative models, operate in ways that are difficult for clinicians to interpret. Black-box recommendations can weaken trust, complicate accountability and make errors harder to detect. The review argues that explainable AI may help, but it is not a complete solution. Systems must also communicate uncertainty, allow clinician review and support clear escalation when predictions are unsafe or unclear.

Workflow integration is equally important. A model that performs well in a retrospective study may fail in practice if it generates alerts at the wrong time, disrupts clinicians, increases documentation workload or contributes to alert fatigue. For that reason, healthcare AI should be tested as part of a broader socio-technical system involving users, patients, workflows, information systems and institutional constraints.

Cybersecurity and privacy risks also rise as AI becomes more embedded in healthcare. AI systems can be exposed to data poisoning, adversarial inputs, model inversion, membership inference and unauthorized extraction of sensitive training information. Because medical data are highly sensitive, the review calls for privacy-preserving methods, access controls, secure storage, audit trails and incident response systems. Federated learning may help by allowing institutions to train models without centralizing raw patient data, but technical safeguards must be paired with governance and oversight.

Regulation and accountability remain unsettled. Many AI tools can change over time through model updates, local recalibration or new data inputs. This makes approval, monitoring and liability more complex than for traditional medical devices. If an AI system contributes to a wrong diagnosis or unsafe treatment recommendation, responsibility may be spread across clinicians, hospitals, vendors, software developers and regulators. The review calls for clear rules defining who approves, deploys, updates, monitors and retires AI systems.

Roadmap calls for lifecycle governance and human oversight

The review provides a governance roadmap for responsible AI integration in healthcare administration and clinical informatics. The roadmap is organized around six stages: data readiness, model validation, workflow integration, governance and accountability, post-deployment monitoring, and workforce readiness.

1. Data readiness: Before AI development, healthcare organizations should assess data completeness, coding consistency, missingness, provenance, population coverage and sources of bias. Reliable AI begins with reliable data. Without representative and well-documented datasets, models may produce unsafe or inequitable results.
2. Model development and validation: AI systems should be tested with clinically relevant metrics, calibration analysis, subgroup performance assessment and robustness checks. External validation is essential because models must work beyond the institution or dataset where they were developed. For high-risk applications, prospective validation should occur before routine clinical use.
3. Workflow integration: AI outputs should be designed around real clinical tasks, decision points and user needs. Alert systems must avoid unnecessary interruptions. Clinicians should understand how AI recommendations fit into care pathways. Human oversight must remain explicit, especially when decisions affect diagnosis, treatment, triage or access to services.
4. Governance and accountability: Institutions need formal structures that include clinicians, informaticians, data scientists, administrators, legal experts, ethicists and patient representatives where appropriate. These structures should define approval processes, error reporting procedures, update rules, patient communication standards and accountability for AI-related decisions.
5. Post-deployment monitoring: AI systems can degrade over time as patient populations, clinical practices, coding patterns, equipment and workflows change. Hospitals must monitor performance, calibration, subgroup outcomes, false positives, false negatives, alert burden, user responses, safety events and signs of data drift. When performance falls, systems may need recalibration, retraining, restriction or withdrawal.
6. Workforce readiness: Clinicians, administrators and health informatics professionals need training in AI literacy, uncertainty, data quality, bias, privacy and safe use of decision support tools. The review argues that AI is unlikely to replace healthcare professionals entirely, but it will reshape roles and responsibilities. The most sustainable model is human-AI collaboration, where automated systems support documentation, triage, prediction and analysis while humans retain responsibility for judgment, communication and ethical decision-making.

The review also looks ahead to future directions, including multimodal AI, generalist medical AI, preventive analytics, precision medicine, edge AI, remote monitoring and global public health surveillance. These tools could support earlier diagnosis, stronger chronic disease management, better hospital planning and faster biomedical discovery. But the paper warns that global benefit will not be automatic.

Low- and middle-income countries face special risks because of weaker digital infrastructure, fragmented health information systems, workforce shortages and under-representation in training datasets. AI systems developed in high-resource settings may not transfer safely without local validation and capacity building. The review asserts that global AI governance must support context-sensitive implementation rather than assuming that imported models will work equally well everywhere.

For healthcare leaders, the findings suggest that AI can improve accessibility, efficiency and quality only when innovation is matched with governance. Without strong validation, real-world monitoring and human oversight, advanced algorithms may add new risks to already strained healthcare systems. With those safeguards in place, AI could become a powerful tool for earlier diagnosis, more personalized treatment, better hospital operations and more equitable patient care.