Detecting mistakes in engineering models: the effects of experimental design
Name
Detecting Mistakes 2nd review v3.pdf
Size
876.26 KB
Format
Adobe PDF
Checksum (MD5)
fb5eaf9fbceb6b41f90fa6c80829bebb
Author(s)
Savoie, Troy B.
Frey, Daniel
Date Issued
November 2011
Journal
Research in Engineering Design
Publisher
Springer-Verlag
Citation
Savoie, Troy B., and Daniel D. Frey. “Detecting Mistakes in Engineering Models: The Effects of Experimental Design.” Research in Engineering Design 23, no. 2 (April 2012): 155–175.
Version
Author's final manuscript
Abstract
This paper presents the results of an experiment with human subjects investigating their ability to discover a mistake in a model used for engineering design. For the purpose of this study, a known mistake was intentionally placed into a model that was to be used by engineers in a design process. The treatment condition was the experimental design that the subjects were asked to use to explore the design alternatives available to them. The engineers in the study were asked to improve the performance of the engineering system and were not informed that there was a mistake intentionally placed in the model. Of the subjects who varied only one-factor-at-a-time, fourteen of the twenty-seven independently identified the mistake during debriefing after the design process. A much lower fraction, one out of twenty-seven engineers, independently identified the mistake during debriefing when they used a fractional factorial experimental design. Regression analysis shows that relevant domain knowledge improved the ability of subjects to discover mistakes in models, but experimental design had a larger effect than domain knowledge in this study. Analysis of video tapes provided additional confirmation as the likelihood of subjects to appear surprised by data from a model was significantly different across the treatment conditions. This experiment suggests that the complexity of factor changes during the design process is a major consideration influencing the ability of engineers to critically assess models.
MIT Department
Massachusetts Institute of Technology. Department of Mechanical Engineering
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1007/s00163-011-0120-y
