Visualizing subjective decision criteria for system evaluation

Abstract

The tools and methods used to evaluate complex systems in terms of human systems integration (HSI) include software, hardware, models, simulations, processes, and techniques. Hundreds of such tools currently exist to evaluate complex HSI systems, and the number is continuously growing. Many HSI tools are developed in research settings, or in an ad-hoc fashion to meet specific project goals. As such, there is no standard repository of HSI tools. Additionally, as HSI systems become more complex and the number of HSI dimensions in which decision-makers evaluate systems continues to grow, the HSI cost-benefit trade space that explores the costs and benefits of a system on the HSI dimensions expands. Therefore, it becomes more difficult for decision-makers to determine if and to what degree a system meets their specific HSI criteria. Engineers, managers, and other decision-makers often have to evaluate complex HSI systems, such as the case of acquisitions in which decision-makers need to rate and compare particular systems. Since there are hundreds of tools that currently exist to evaluate the HSI aspects of systems, decision-makers may not be familiar with all of them. Additionally, with the HSI cost-benefit trade space continually expanding, decision-makers need a way to reduce the number of HSI tools, or downs elect, from the hundreds of HSI tools to a smaller set of desirable tools. This smaller set of desirable tools is defined by users' criteria, which is often vague and ambiguous. This research investigates the visualization techniques of such downselection tools and how the visual representation of information impacts the decisions of decision-makers. The results of this work show that salient information biases decision-makers' decisions when using decision support tools (DSTs). Also, information aggregation did not appear to have an effect on decision-makers' level of automation bias when using decision support tools. There did not appear to be any correlation between decision-makers' trust in DSTs and the extent to which they rely on the results presented in those tools. Additionally, decision-makers' perception of the ease of use of DSTs is correlated with their trust in those tools, although there appeared to be an influence of age on users' perceptions of such tools. While this thesis focuses on the specific example of downselecting a database of HSI tools for a decision maker, the overall processes and research in this effort can generally be applied to any downselection from a large database of elements to a smaller set of desirable elements using both objective and subjective decision criteria of decision-makers.