Pharma industry must adopt human-in-the-loop AI to meet strict validation protocols

The study makes clear that regulators expect human oversight to remain central to pharmaceutical quality assurance. Human operators must participate at key control points and retain authority to review, approve or override AI-generated insights. AI is not permitted to replace human accountability in GMP processes.

The pharmaceutical sector is entering a decisive moment in its digital transformation as regulators move to define how artificial intelligence can be deployed in manufacturing. A new scientific review warns that without strict human oversight and rigorous technical controls, AI-driven systems used in production could compromise product quality and patient safety. The study argues that the next phase of pharmaceutical modernization depends on the ability to combine data-driven intelligence with human judgment under a clear regulatory framework.

The study, titled Human-in-the-Loop AI Use in Ongoing Process Verification in the Pharmaceutical Industry and published in Information, analyzes how AI can be integrated into Ongoing Process Verification (OPV) while complying with Good Manufacturing Practices and the European Union’s recently introduced Annex 22. The review lays out technological, regulatory and operational considerations for embedding AI safely into critical manufacturing environments.

Regulatory pressure forces industry to rethink AI use

Drug manufacturing is shifting away from periodic validation checkpoints toward continuous process verification driven by real-time data. This shift makes AI an attractive tool for identifying subtle process variations, predicting failures, and improving consistency across large-scale operations. AI has already demonstrated value in multivariate process analysis, anomaly detection, dynamic control decision systems, and predictive maintenance.

However, the industry now faces a regulatory landscape that sharply limits how AI may be used. Annex 22, the European Union’s new framework for AI in medicinal product manufacturing, is described in the review as one of the most restrictive regulatory positions globally. It allows only deterministic, non-adaptive AI systems to operate in GMP-critical decision chains. Deterministic models produce the same output for the same input, making their behavior predictable, testable and auditable. These properties are considered essential for ensuring product safety and regulatory compliance.

Adaptive, probabilistic, or generative AI models, including large language models and neural networks capable of changing behavior after deployment, are prohibited from making or executing decisions that affect product quality. They may only be used in supportive roles such as pattern recognition or as advisory systems, and only if human operators maintain full decision authority.

This regulatory posture is intended to guard against unpredictable AI behavior and lack of explainability, which could compromise pharmacovigilance, product integrity or patient safety. At the same time, it has introduced operational uncertainty for manufacturers seeking to modernize production lines. The review stresses that companies must redesign their digital strategies to align with Annex 22’s requirements.

The study makes clear that regulators expect human oversight to remain central to pharmaceutical quality assurance. Human operators must participate at key control points and retain authority to review, approve or override AI-generated insights. AI is not permitted to replace human accountability in GMP processes.

Human-in-the-loop systems emerge as the only compliant path forward

Human-in-the-Loop (HITL) architectures offer the clearest path to compliant AI integration. These systems pair computational intelligence with structured human review, enabling manufacturers to benefit from AI’s analytical speed while maintaining regulatory-required human control.

The review outlines a standard HITL model consisting of data acquisition layers, multivariate feature processing modules, deterministic AI engines, human decision interfaces, and execution controls. This architecture ensures that AI functions as an assistant rather than an autonomous actor. Human reviewers evaluate AI insights, validate recommendations and make the final decisions that affect process adjustments.

Explainability tools such as SHAP and LIME are identified as critical components. Regulators require that human operators understand why an AI system flags a deviation or suggests an intervention. Explainability also supports operator training, deviation investigations and quality management audits. Without clear interpretability, the AI system cannot be considered compliant.

The authors highlight the importance of traceability mechanisms that capture all data inputs, model outputs, operator interactions and final decisions. These traces must be auditable and tamper-resistant to comply with regulatory standards. The review notes that Annex 22 mandates extensive documentation for AI systems, including design justification, validation protocols, lifecycle maintenance procedures, risk assessments and monitoring plans.

A strong emphasis is placed on data governance. AI performance depends heavily on the availability and cleanliness of manufacturing data. Poor data quality, incomplete datasets or unmonitored drift can undermine model reliability. Therefore, effective HITL systems require robust data management policies, access controls, integrity checks and continuous training for staff.

Through case studies, the paper shows how HITL frameworks have already begun to enhance quality assurance. Examples include real-time monitoring of granulation processes, early detection of filling anomalies in sterile environments and accelerated root-cause analysis for deviations. These deployments demonstrate shorter investigation times, improved process consistency and reductions in costly deviations.

AI deployment requires strict training, validation and continuous monitoring

The review presents a detailed roadmap for companies seeking to implement HITL AI in OPV. The first stage involves conducting feasibility studies to determine whether specific processes can benefit from AI without violating Annex 22. Manufacturers must identify where AI can provide value and where traditional statistical methods remain more appropriate.

The next phase focuses on model development and validation. Deterministic models must undergo extensive testing across representative datasets before they are deployed. Validation must assess accuracy, robustness, reproducibility, and failure behavior under stress scenarios. Once validated, models must remain static, as post-deployment learning is not permitted in GMP-critical systems.

Human oversight protocols must be established before an AI system goes live. These include defining intervention thresholds, designing operator dashboards, establishing alert workflows and specifying who has decision authority at each stage of the process. Human operators must be trained not only to interpret AI outputs but to understand the limitations of the models.

The authors highlight that HITL systems require continuous monitoring to detect model degradation, input anomalies or shifts in process behavior. Even deterministic models can fail if input data changes or if upstream processes introduce new variables. Monitoring frameworks must capture performance metrics, incident logs and user interactions, and must be supported by structured review cycles.

The study notes that while HITL systems create additional procedural layers, they also reduce operational risk. Human involvement prevents overreliance on automated decisions and ensures that deviations are interpreted within proper context. The combination of AI precision and human domain expertise leads to improved process understanding and more resilient decision-making.

The review also outlines the limitations of HITL AI deployment. Implementation costs can be high, data availability may be insufficient, and workforce training gaps can delay adoption. Some AI methods with high predictive capability, such as deep learning, cannot be used in GMP-critical environments due to Annex 22 restrictions. This reduces the range of tools available to manufacturers. Despite these challenges, the authors argue that HITL approaches remain the safest and most viable path for integrating AI into pharmaceutical OPV.

The paper calls for industry-wide collaboration to refine standards, develop compliant models, expand operator training and share best practices. The authors argue that as digital transformation accelerates, companies that successfully integrate AI under HITL conditions will gain competitive advantages in quality, efficiency and regulatory preparedness.