Designing a graphical decision support tool to improve system acquisition decisions

Abstract

System acquisition decision makers are frequently charged with choosing a single system from a set of feasible possibilities that could best fulfill the needs of their organizations. While numerous rules and regulations are already in place for both commercial and government acquisitions to ensure the acquisitions are conducted fairly, decision makers need greater support than rules and regulations alone can provide. The acquisition decision is a complex data analysis problem, where the decision maker must analyze multiple candidate systems on a number of performance and cost metrics. To understand this multivariate environment, decision makers must analyze the system data at multiple levels of reasoning. This research proposes a decision support tool that best supports system acquisition decision makers by providing them with graphical representations displaying how well candidate systems fulfill their organizations' needs. System acquisition decisions require support of three basic levels of reasoning (Data Processing, Information Aggregation, and Knowledge Synthesis) in order to perform system trade-offs on relevant system metrics. To test how well decision support tools could support system acquisition decision makers, two graphical decision support tools were designed: a traditional separable display and a new configural display named Fan Visualization (FanVis). To compare the effectiveness of FanVis against a traditional separable display, an experiment was conducted where participants answered a series of system acquisition questions across the three levels of reasoning.

(cont.) Analysis of the experimental results indicate that FanVis and the separable displays support a system acquisition decision maker, but to different degrees across the three levels of reasoning. Comparatively, participants tended to have higher performance on Knowledge Synthesis tasks using FanVis, while they tended to have a higher performance on Data Processing tasks using the separable display. When examining subjective measures, FanVis was the preferred tool of choice. Through use of an eye tracking device, it was further determined that participants also exhibited erratic fixation patterns on those questions that were answered incorrectly compared to those answered correctly. Further, it was determined that FanVis allowed participants to maintain more efficient gaze patterns regardless of task, whereas participants used less efficient gaze patterns in the separable display for some tasks. Additionally, participants tended to spend a greater frequency of time fixating on relevant elements in FanVis while completing Knowledge Synthesis tasks, while the opposite was true for Data Processing tasks, suggesting that performance and time spent fixating on relevant information is correlated. From the results of this experiment, a set of design implications was created for future system acquisition decision support tools.