Verification Planning for Efficient Uncertainty Reduction of Space Science Systems

Abstract

Verification for a complex engineering system is essential for mitigating risks associated with quantitative uncertainty and ensuring confidence in the system’s performance. Verification planning is the problem of deciding how to allocate resources in this process, and verification for some systems may allocate more time and money to certain verification activities than is necessary to achieve a desired level of certainty. The need for efficient and effective verification is especially great for space systems and space science instruments, which are often subject to active cost and schedule constraints that make verification planning a challenge. This research proposes the Uncertainty Quantification Verification Planning Methodology (UQVM) for designing optimal-under-uncertainty verification plans in a systematic, quantitative, model-based way. Uncertainty quantification methods are used to model instrument performance, determine sources and magnitudes of parametric uncertainty, and perform sensitivity analysis. Stochastic models of potential verification activities are also developed with subject matter expertise, and are subjected to uncertainty quantification. A novel approach to optimal Bayesian experimental design (OBED) is developed to determine sets of verification activities that minimize effort and maximize certainty of system performance. A comprehensive systems engineering approach brings together these techniques of uncertainty quantification and experimental design methods, so that systems engineers can optimize design of AI&T and V&V campaigns with respect to programmatic cost and confidence of system performance, and can refine those plans as new verification data is obtained. UQVM is demonstrated for space science system case studies. A study of verification planning for CCD performance shows that optimally uncertainty-reducing plans within a cost cap can be identified that reduce the variance of predicted performance by 67%, and that iterative data-informed verification planning can reduce the variance of predicted performance by 96%. A retrospective analysis of the sensitivity verification for the Large Lenslet Array Magellan Spectrograph (LLAMAS) shows that optimized plans can reduce testbed time by 94% without a loss in uncertainty reduction, and that plans can be identified that perform better than historical plans with a confidence of greater than 90%. An early-phase analysis of precision control verification for the James Webb Space Telescope (JWST) shows that tests can be ranked in order of benefit-at-cost.