Towards measuring attention allocation in model-based engineering teamwork

Abstract

Large organizations succeed when they are operating with systemic awareness. A similar dynamic could be in effect at the level of teams and teams of teams within engineering organizations. Engineering teams are tasked with solving problems of multidisciplinary nature with multiple stakeholder constraints. For better performance, teams need to be aware of multiple constraints simultaneously. This thesis explores the design of sensors to measure team awareness using team attention in the problem and solution spaces. The concept of situational awareness and attention allocation has been studied in literature in the context of user interaction design for individuals or teams in areas where team members could be overwhelmed with information. Engineering problem solving is of a similar nature where team members have to be aware of multiple information sources and bring their attention to the right pieces of information to make decisions. To define what is "right", the concept of a problem space and solution space is defined. It is hypothesized that higher performing teams allocate their attention to systemically significant portions of the problem and solution spaces. An experiment is designed assuming that short strategy discussions are result in greater attention allocation to systemically significant portions of the problem and solution spaces. The concept of using a strategy discussion to spur creativity was based on literature in team creativity and innovation which posited that a process consisting of iterations of first divergent and then convergent thinking results in greater innovation. A model or toy problem of designing an innovation campus is chosen. Data from 50 teams spending one hour with a model exploration interface are analyzed. This problem was designed to be easily understandable in a short time by conference-goers from which a pool of volunteer participants were assigned randomly to 50 teams of 2 to 4 participants per team. Out to 50 teams, 14 teams were assigned to a control group who were instructed to perform a placebo discussion instead of a strategy discussion. Two other groups consisting of 14 and 15 teams respectively had strategy discussions at the beginning and middle of the experiment time-slot. Pareto frontier based ranking methods are used to rank team performance. The performance across different groups are compared using hypothesis testing methods. Results suggest that strategy discussions help to arrive at more effective problem-solving. These results are not statistically significant at 95% confidence level (they were significant at 85% confidence level) using a Kruskal-Wallis hypothesis, they do provide promising directions for further work. Besides directly testing the hypothesis, other observations were made on the data. One interesting observation was that teams that had strategy discussions tended to perform better as they executed more simulations, while teams that had the placebo discussion tended to perform worse as they executed more simulations. The data gathered during the experiments has not been fully analyzed due to the scope of the thesis. Further work that could be done include analyzing user interface "fingerprints" to measure attention allocation directly to test the assumption that strategy discussion during a decision making session results in higher attention allocation to systemically significant portions of the problem and solution spaces. An attempt is made at defining the concept of attention allocation, and quantitatively measuring how much of attention is allowed to systemically significant portions of the problem and solution spaces. Further work is also warranted in exploring alternate definitions and calculation of this metric.