If a Picture is Worth 1000 Words, Is a Word Worth 1000 Features for Design Metric Estimation?
Name
md_144_4_041402.pdf
Description
Published version
Size
576.12 KB
Format
Adobe PDF
Checksum (MD5)
eb35d079f46fc6373c7c4f4a478076b2
Author(s)
Edwards, Kristen M
Peng, Aoran
Miller, Scarlett R
Ahmed, Faez
Date Issued
April 1, 2022
Journal
Journal of Mechanical Design
Publisher
Wiley
Citation
Edwards, K. M., Peng, A., Miller, S. R., and Ahmed, F. (December 17, 2021). "If a Picture is Worth 1000 Words, Is a Word Worth 1000 Features for Design Metric Estimation?." ASME. J. Mech. Des. April 2022; 144(4): 041402.
Version
Final published version
Abstract
Abstract
A picture is worth a thousand words, and in design metric estimation, a word may be worth a thousand features. Pictures are awarded this worth because they can encode a plethora of information. When evaluating designs, we aim to capture a range of information as well, including usefulness, uniqueness, and novelty of a design. The subjective nature of these concepts makes their evaluation difficult. Still, many attempts have been made and metrics developed to do so, because design evaluation is integral to the creation of novel solutions. The most common metrics used are the consensual assessment technique (CAT) and the Shah, Vargas-Hernandez, and Smith (SVS) method. While CAT is accurate and often regarded as the “gold standard,” it relies on using expert ratings, making CAT expensive and time-consuming. Comparatively, SVS is less resource-demanding, but often criticized as lacking sensitivity and accuracy. We utilize the complementary strengths of both methods through machine learning. This study investigates the possibility of using machine learning to predict expert creativity assessments from more accessible nonexpert survey results. The SVS method results in a text-rich dataset about a design. We utilize these textual design representations and the deep semantic relationships that words and sentences encode to predict more desirable design metrics, including CAT metrics. We demonstrate the ability of machine learning models to predict design metrics from the design itself and SVS survey information. We show that incorporating natural language processing (NLP) improves prediction results across design metrics, and that clear distinctions in the predictability of certain metrics exist. Our code and additional information about our work are available on the MIT DeCoDE Lab website.1
A picture is worth a thousand words, and in design metric estimation, a word may be worth a thousand features. Pictures are awarded this worth because they can encode a plethora of information. When evaluating designs, we aim to capture a range of information as well, including usefulness, uniqueness, and novelty of a design. The subjective nature of these concepts makes their evaluation difficult. Still, many attempts have been made and metrics developed to do so, because design evaluation is integral to the creation of novel solutions. The most common metrics used are the consensual assessment technique (CAT) and the Shah, Vargas-Hernandez, and Smith (SVS) method. While CAT is accurate and often regarded as the “gold standard,” it relies on using expert ratings, making CAT expensive and time-consuming. Comparatively, SVS is less resource-demanding, but often criticized as lacking sensitivity and accuracy. We utilize the complementary strengths of both methods through machine learning. This study investigates the possibility of using machine learning to predict expert creativity assessments from more accessible nonexpert survey results. The SVS method results in a text-rich dataset about a design. We utilize these textual design representations and the deep semantic relationships that words and sentences encode to predict more desirable design metrics, including CAT metrics. We demonstrate the ability of machine learning models to predict design metrics from the design itself and SVS survey information. We show that incorporating natural language processing (NLP) improves prediction results across design metrics, and that clear distinctions in the predictability of certain metrics exist. Our code and additional information about our work are available on the MIT DeCoDE Lab website.1
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1115/1.4053130
