The Secret to Safer AI May Lie in Decades of Human Safety Experience

Artificial intelligence is entering environments where a wrong decision can compromise critical infrastructure, disrupt essential operations or endanger human lives. Still, the methods used to establish confidence in conventional software are struggling to address systems that learn from data, behave probabilistically and encounter conditions their developers may never have anticipated.

A study titled "Assurance of Safety-Critical AI-Enabled Capabilities: Integrating Human, Software, and Machine Learning Assurance Paradigms," published in the journal AI, challenges the assumption that stronger algorithms and conventional testing alone can establish dependable artificial intelligence. Authored by Randall McCutcheon, Keith F. Joiner, Li Qiao and John Harvey, the research proposes a framework for integrating technical validation, organisational oversight and human accountability.

The researchers introduce 20 assurance precepts and a Dual Assurance Spiral for AI-enabled Capabilities (DAS4AIC), designed to organise safety responsibilities throughout development and operation. Their central proposition is that decades of experience managing human performance and software reliability offer valuable foundations for governing increasingly intelligent systems.

The Safety Problem Hidden Behind AI's Technical Performance

Conventional software engineering establishes confidence by comparing system behaviour against defined requirements. Engineers examine whether programs execute intended functions, satisfy safety constraints and respond predictably under specified conditions. Machine-learning systems complicate this approach because their behaviour emerges from training data, model architecture and optimisation rather than explicitly programmed instructions alone.

An AI model can perform reliably during development yet encounter unfamiliar inputs after deployment. Differences between training and operational environments, unexpected interactions among variables and changing data patterns can produce behaviour that conventional verification fails to anticipate. High performance under controlled conditions therefore offers incomplete evidence of operational safety.

The study identifies data dependency, sensitivity to unfamiliar conditions, limited explainability, emergent behaviour and lifecycle evolution as distinctive assurance challenges. These characteristics require organisations to investigate not only whether a model produces correct outputs, but whether its underlying evidence, operating boundaries and failure responses are sufficiently understood.

The researchers examine three overlapping traditions of assurance. Human assurance focuses on anticipating failures, maintaining situational awareness and managing organisational variability. Software assurance emphasises requirements, verification, traceability and evidence-based confidence. AI assurance must integrate both approaches while addressing uncertainty introduced by machine learning.

Their starting proposition is deliberately provocative: if organisations can manage human performance in safety-critical environments, some of those management principles may also help assure AI. The analogy does not assign human characteristics or accountability to machines. Instead, it recognises similarities involving training, performance variability, monitoring, communication and adaptation.

Sixty Safety Principles Reveal an Unexpected Link Between Humans and Machines

To investigate these relationships, the researchers derived 20 assurance precepts for each of the three domains, creating 60 principles for comparison. Their selection drew on foundational safety and engineering literature, expert judgement and author verification, with Microsoft Copilot assisting the initial extraction of candidate precepts.

A workshop involving 14 participants from project management, finance, information technology and other professional backgrounds provided preliminary conceptual validation. Participants examined the proposed principles and identified relationships between human and software assurance, followed by software and AI assurance.

The mapping connected 13 human assurance precepts with 13 software assurance precepts. In contrast, all 20 software assurance precepts were connected to 19 of the 20 AI assurance precepts. The findings indicate substantial continuity between established software engineering practices and emerging approaches to machine-learning safety.

One AI precept involving representative users throughout planning, development and testing had no identified software connection during the workshop. The researchers acknowledge that related software principles exist, but participants did not establish those connections in the exercise. The result highlights how human participation can become less visible within technically oriented assurance processes.

A subsequent mapping undertaken by the authors suggested that some human assurance practices connect more directly with AI than through conventional software engineering. Operational monitoring was particularly prominent, linking to 12 human assurance precepts. Practices involving organisational safeguards, anticipation of failure and ongoing observation also showed substantial relevance.

However, the direct human-to-AI mapping was not independently validated and used a different process from the workshop. The authors caution that stronger apparent connections may partly reflect methodological differences. Their findings establish conceptual relationships, not proof that human-management techniques automatically improve AI safety.

The resulting 20 AI assurance precepts cover four connected areas: system risks and human roles; data, modelling and performance measures; uncertainty verification and stress testing; and operational monitoring, deployment and structured safety evidence.

A particularly important contribution concerns training data. The researchers argue that datasets must represent operationally meaningful factors, including combinations capable of producing rare but consequential failures. Randomly separating available observations into training and testing datasets cannot, by itself, establish whether the model has encountered the conditions most relevant to safety.

They also question reliance on overall accuracy. Different prediction errors can produce markedly different operational consequences, meaning performance measures and optimisation objectives must reflect system-specific risks. Comparing AI against established operational baselines becomes essential when deciding whether deployment represents a genuine safety improvement.

Beyond the Algorithm: A Dual Assurance Spiral Puts Accountability Into Engineering

The proposed DAS4AIC framework attempts to address a persistent weakness in AI governance: broad safety principles frequently identify desirable outcomes without fully establishing who should deliver them, when assurance activities should occur or how responsibilities should be coordinated.

Its first spiral, Architecture Selection and Feasibility, examines whether AI is appropriate for a proposed application. It considers alternative architectures, operational hazards, initial safeguards and the relationship between automated functions and human authority.

The second, Architecture Application and Qualification, concentrates on implementing the selected architecture, generating evidence, testing performance and establishing confidence in continued operation. Both spirals are intended to operate iteratively rather than represent separate, one-time approvals.

Four recurring perspectives connect the framework: human–autonomy teaming, operational challenge, causal representativeness, and explainability and causal fidelity. Together, they require organisations to assess whether people can safely operate alongside AI, whether systems withstand realistic challenges, whether training reflects operational conditions and whether model behaviour can be adequately explained.

The framework assigns activities to relevant specialists, including operators, trainers, human-factors experts, safety engineers, operational testers, domain experts and certification authorities. A continuing assurance record preserves safety arguments, identified hazards, assumptions, supporting evidence, retraining triggers and decisions about renewed qualification.

Human–autonomy teaming receives particular attention. The study proposes explicit trust contracts defining when AI may act independently, when uncertainty must be communicated and when human intervention or escalation is required. These arrangements seek to prevent excessive reliance on automation without unnecessarily restricting useful capabilities.

Testing must therefore evaluate the combined human-machine system, including responses under pressure, degraded conditions and unexpected events. Operators need opportunities to interpret, challenge and override automated outputs. The researchers also warn that increasing automation can erode human understanding and oversight capability unless organisations maintain collective training and realistic exercises.

The framework additionally treats model updates as controlled safety decisions. Performance deterioration, unfamiliar conditions or accumulated exceptions may justify retraining, but revised models require renewed verification and validation. An updated algorithm cannot automatically inherit confidence established for an earlier configuration.

Unlike methodologies primarily focused on constructing machine-learning safety cases, DAS4AIC is intended to coordinate assurance responsibilities across the wider capability lifecycle. The authors present it as complementary to existing approaches rather than a replacement for established engineering practices.

The Governance Test: Stronger Safety Architecture Still Needs Real-World Proof

The research carries practical relevance for institutions integrating AI into aviation, defence, healthcare, cybersecurity and infrastructure. Its approach places responsibility on organisations to maintain continuing evidence of safety, rather than depending exclusively on successful initial testing or claims of algorithmic accuracy.

For capability managers and system developers, this means involving operational users, testing specialists and training organisations earlier in development. Decisions concerning data selection, performance measures, model architecture and human intervention should reflect anticipated operating hazards before a system enters service.

The proposed framework also offers a structured way to connect ethical expectations with measurable engineering activities. Transparency, accountability and trustworthy operation become questions of documented evidence, allocated responsibility and demonstrated performance rather than abstract commitments alone.

Nevertheless, the evidence supporting the framework remains preliminary. The 14-person workshop was weighted toward project management rather than specialist safety engineering or AI disciplines. Participants were recruited through an assurance masterclass, and workshop findings relied on author-generated discussion transcripts without independent participant review or inter-rater reliability assessment.

The selection of assurance precepts also involved expert judgement and a limited literature base rather than systematic thematic analysis. More importantly, the authors have not demonstrated DAS4AIC through operational case studies, practical trials or comparative testing against established assurance methodologies.

Consequently, the research does not establish that its framework reduces failures, improves safety outcomes or makes certification more efficient. Its contribution lies in developing a coherent conceptual architecture whose practical effectiveness remains to be tested.

The researchers identify worked examples, case studies and trials as priorities, alongside more rigorous assessment of relationships between assurance traditions. Further investigation must also examine operational accountability, particularly where responsibility is distributed across developers, integrators, operators and organisational leadership.

The deeper finding concerns the nature of trust in intelligent systems. Reliable AI cannot be established solely through model performance because safety depends on the interaction between algorithms, operational environments, institutional procedures and human judgement.