Using STPA to inform developmental product testing

Abstract

Developmental product testing currently evaluates system safety the same way it evaluates system performance: it attempts to isolate individual components' behaviors to evaluate their reliability. However, today's systems are often irreducible because of their complexity, leaving current practices ineffective at identifying safety deficiencies. Evolving to a modern systems-based hazard analysis is important for product development. Products stand to benefit during the testing stage, before initial fielding. In test, designs meet operation for the first time, and use practices and organizational influences both contribute to the safety of the system. By evaluating safety as an emergent property, hazards that emerge because of the testing process itself can be mitigated, and hazards that exist because of the inherent system design and use philosophy can be identified and traced throughout development and fielding. System-Theoretic Process Analysis (STPA), developed by Nancy Leveson at the Massachusetts Institute of Technology, is a modern hazard analysis technique that identifies unsafe scenarios in a system in order to generate requirements to eliminate or control those scenarios. It improves on traditional reductionist approaches that treat accident causation only as a linear chain of events or probabilistic occurrence of simultaneous component failures (including human error). While systems-based and complete, STPA could benefit from additional guidance, particularly in the identification of human contributions to accidents. The present research begins by extending STPA to include more guidance for the controller analysis, including refinements to the process model, fundamental human-engineering considerations, and socio-organizational influences. Next, Leveson's organizational control structure example is updated to include a test stage that serves as an intermediary between design and field use. Model inclusion criteria are updated, and Explicit-Influence Maps are introduced as a tool to understand the organization and aid in hazard analysis. Finally, this research investigates the U.S. Air Force developmental testing enterprise and applies STPA to a product test. Results are compared to that of the test-safety planning and reporting techniques traditionally in use, and utility is assessed with a research survey administered to developmental test professionals. Keywords: STAMP, STPA, system safety, hazard analysis, product testing, test safety, problem reporting, safety certification