Healthcare AI requires prospective testing and risk monitoring

Researchers have warned that the way artificial intelligence-powered healthcare tools are evaluated may be dangerously incomplete. In a detailed new review published in the journal Diagnostics, a team of researchers notes that diagnostic AI must be assessed through a clinically grounded, risk-aware framework rather than narrow accuracy benchmarks.

Their study, titled “TRIAGE: Trustworthy Reporting and Assessment for Clinical Gain and Effectiveness of AI Models,” introduces a structured evaluation framework designed to close what the authors describe as a widening gap between reported model performance and the level of evidence required for safe clinical adoption.

From accuracy to clinical accountability

According to the authors, much of the current diagnostic AI literature relies heavily on retrospective validation and single summary metrics such as accuracy or area under the curve. While such measures provide insight into discrimination performance, they do not fully capture clinical risk, fairness implications, threshold selection trade-offs or deployment constraints.

The TRIAGE framework organizes diagnostic AI evaluation around four primary use cases: screening systems, triage tools, second-reading support systems and confirmatory diagnostic decision aids. Each use case demands different evidence standards. A screening model, for example, must prioritize sensitivity to avoid missing disease, while maintaining a clinically manageable false-positive burden. A confirmatory diagnostic system, by contrast, requires stronger external validation, clearly defined reference standards and demonstration that its use improves decisions without introducing unacceptable harm.

Central to the framework is a detailed examination of discrimination metrics derived from the confusion matrix. Sensitivity, specificity, positive and negative predictive values, likelihood ratios, diagnostic odds ratio and F-scores are treated not as interchangeable statistics but as measures that carry distinct clinical meaning. The authors highlight that predictive values are highly dependent on disease prevalence and spectrum effects. A model trained in a high-prevalence hospital setting may yield misleading predictive performance when deployed in community screening.

The paper also addresses multi-class and multi-label diagnostic settings. The authors recommend appropriate aggregation strategies such as micro, macro and weighted averaging, ensuring that performance reporting reflects the distribution of disease categories. They also discuss set-based measures such as Hamming loss, exact match ratio and Jaccard similarity variants, which are particularly relevant when multiple diagnoses or conditions are predicted simultaneously.

The study further integrates threshold-dependent analysis into the evaluation process. Representation curves such as ROC and precision–recall curves are discussed alongside calibration assessment and decision-curve analysis. Calibration slope and Brier score are presented as critical tools when AI outputs represent risk probabilities rather than simple labels. In this framework, selecting a decision threshold is not a purely statistical exercise but a clinical judgment that balances the harms of false positives and false negatives.

Robustness, fairness and validation integrity

A defining feature of the TRIAGE framework is its insistence that diagnostic AI must be tested for robustness and fairness before being deemed clinically credible.

Robustness is defined operationally as stability under controlled perturbations. In real-world clinical environments, measurements are subject to device variability, patient movement and recording noise. The authors recommend one-at-a-time perturbation testing, where incremental noise is injected into individual features while monitoring degradation in performance metrics such as sensitivity or F1-score. Models that degrade sharply under minor perturbations are flagged as potentially unreliable in operational settings.

Fairness is addressed through a structured bias taxonomy. The paper identifies representative bias, which occurs when certain populations are underrepresented in training data; selective bias, which emerges from restrictive inclusion criteria; and measurement bias, where variables are recorded differently across demographic groups. To operationalize fairness evaluation, the authors adopt equalized odds as a criterion. This requires that true positive rates and false positive rates be comparable across sensitive attributes such as sex or age group. Under this principle, a model that performs well overall but systematically under-detects disease in a minority population would fail fairness evaluation.

Validation design is treated as equally critical. The framework outlines multiple cross-validation strategies and their appropriate use. Stratified cross-validation is recommended when class imbalance is significant, ensuring that disease prevalence is preserved in each fold. Grouped cross-validation is essential when multiple samples originate from the same patient, preventing leakage between training and test sets. Temporal or time-series cross-validation is required for longitudinal clinical data, ensuring that future information is not inadvertently used to predict past outcomes.

The authors caution against common methodological pitfalls. Paired Student’s t-tests may be inappropriate when cross-validation introduces dependence between folds. Alternative statistical methods such as Dietterich’s 5×2 cross-validation approach and Fisher’s Exact Test are discussed for rigorous model comparison. The paper also highlights the importance of multiplicity control when testing multiple subgroups or metrics to prevent inflated claims of significance.

Agreement metrics such as Cohen’s kappa are introduced for evaluating inter-rater reliability and model agreement beyond chance, particularly in datasets labeled by multiple clinicians. These tools help ensure that AI performance is interpreted in the context of human variability.

Deployment readiness, energy costs and governance

The TRIAGE framework extends beyond performance statistics to consider operational feasibility and governance alignment.

Energy consumption is introduced as an often-overlooked dimension of AI evaluation. Deep learning systems can require substantial computational resources for training and inference, contributing to infrastructure costs and environmental impact. The authors recommend reporting energy use in joules or watt-hours and considering performance-per-watt as part of system assessment. Optimized algorithms, efficient hardware and sustainable energy sources are framed as components of responsible AI deployment.

The study distinguishes between retrospective validation and prospective evaluation. Retrospective testing, while necessary, is insufficient to guarantee clinical reliability. Real-world deployment occurs under time pressure, incomplete information and shifting patient populations. Silent deployment, where predictions are generated without influencing care, can detect calibration drift and distribution shift. Interventional prospective trials can measure how AI systems affect clinician decisions, diagnostic timing and patient outcomes.

Clinical utility is defined not solely by diagnostic accuracy but by whether AI integration improves decision-making without introducing new harms such as automation bias, alert fatigue or workflow overload. The authors stress that diagnostic AI evaluation must ultimately answer a safety question: does the evidence support use without unacceptable harm?

External validation across institutions is presented as a primary safety requirement rather than an optional enhancement. Single-center performance may reflect site-specific artifacts rather than disease-related signal. Multicenter and temporal validation help demonstrate generalizability and guard against silent performance degradation over time.

The framework aligns its recommendations with international governance and risk management standards. It reflects principles found in global guidance documents emphasizing lifecycle monitoring, accountability and continuous oversight. Monitoring is framed as an institutional responsibility, requiring documentation, corrective action triggers and structured reporting.