Informing decision-making in single-objective, mixed-variable design problems

Abstract

Data-driven decision-making in mixed-variable design problems presents a variety of challenges and opportunities, especially in the increasingly data-rich field of emissions in architectural and structural design. Designers can benefit from an underlying knowledge about, for example, whether material choice (discrete) or span (continuous) have more important consequences on structural emissions. This intuition need not be built purely through experience nor optimization: data-driven approaches can offer quantitative feedback. However, traditional approaches of sensitivity analysis are limited to continuous variables, while certain types of machine learning models can handle combinations of continuous and discrete variables. In this thesis, a hybrid gradient-based, sampling-based technique determining the directional importance of mixed variables in a design space is benchmarked against state-of-the-art variable importance methods (also known as feature importance or interpretability) from machine learning. The importance evaluations and runtimes are compared across workflows. First, a concise literature review is presented, clarifying and unifying terminology across fields. Tree-based models are identified as a machine learning model that readily handles mixed-variable design spaces, and the following variable importance metrics are identified: impurity-based importance metrics (also known as Mean Decrease Impurity), permutation feature importance (PFI, also known as Mean Decrease Accuracy), and Shapley values. These existing workflows are applied to varying sample sizes of three different datasets related to the application to low-carbon structural design. The same samples are evaluated using the hybrid technique previously proposed by the author, which trains the data on a conditional variational autoencoder (cVAE), approximates gradients on the model, and summarizes gradients into “influence metrics” using a Gaussian mixture model (GMM) (in contrast to a mean absolute value). Through this comparison, this thesis establishes several findings, including several advantages to using the hybrid cVAE and GMM-to-influence workflow over typical tree-based feature importance approaches. First, the hybrid method’s evaluation of gradients is consistently faster than the evaluation of importance in all other workflows for all sample sizes and datasets. Secondly, it avoids the known drawback of tree-based models’ tendency to assign higher importance to high-cardinality variables. Third, its definition of performance “gradients” with respect to each category (as opposed to each categorical variable) offers more specific, useful insights. For example, it is more useful to know which structural framing system is associated with large reductions in emissions (gradients by category) than to know that the choice of structural framing system is associated with a range of reductions and increases in emissions (gradients by categorical variable, which is typical in feature importance methods). These advantages come at the expense of more time (in this case, 10-fold) needed to train the model compared to state-of-the-art gradient-boosted tree models and the additional time needed to fit a GMM (as opposed to taking the mean absolute value of importance values across the sample). The hybrid workflow is still 2 to 10 times faster than the random forest workflows. Finally, these comparisons highlight the importance of cardinality of categorical variables in mixed variable design spaces, both in the process of selecting a model and selecting an importance evaluation method. Key words: variable importance, feature importance, mixed-variable design spaces, gradients, design space exploration, data-driven decision-making