Ensembles of Adaptive One-Factor at-a-Time experiments : methods, evaluation, and theory

Abstract

This thesis recommends an experimentation methodology which can be used to improve systems, processes and products. The proposed technique borrows insights from statistical prediction practices referred to as Ensemble Methods, to extend Adaptive One-Factor-at-a-Time (aOFAT) experimentation. The algorithm is developed for an input space where each variable assumes two or more discrete levels. Ensemble methods are common data mining procedures in which a set of similar predictors is created and the overall prediction is achieved through the aggregation of these units. In a methodologically similar way this study proposes to plan and execute multiple aOFAT experiments on the same system with minor differences in experimental setup, such as starting points, or order of variable changes. Experimental conclusions are arrived at by aggregating the multiple, individual aOFATs. Different strategies for selecting starting points, order of variable changes, and aggregation techniques are explored. The proposed algorithm is compared to the performance of a traditional form of experimentation, namely a single orthogonal array (full and fractional factorial designs), which is equally resource intensive. Comparisons between the two experimental algorithms are conducted using a hierarchical probability meta-model (HPM) and an illustrative case study. The case is a wet clutch system with the goal of minimizing drag torque. Across both studies (HPM and case study), it is found that the proposed procedure is superior in performance to the traditional method.

(cont.) This is consistent across various levels of experimental error, comparisons at different resource intensities, and through a wide array of response surfaces generated by the meta-model. At best, the proposed algorithm provides an expected value of improvement that is 15% higher than the traditional approach, at worst, the two methods are equally effective, and on average the improvement is about 10% higher. These findings suggest that the ensemble of aOFATs can be an effective and often preferred alternative to the use of orthogonal arrays for experimentation. This thesis also shows that it more effective to apply ensemble procedures to aOFAT versus applying ensemble techniques on multiple, highly-fractioned orthogonal designs (each being as resource intensive as a single aOFAT). A theoretical discussion explaining the reasons for the superior performance of the proposed algorithm supports the empirical findings.