2026-09-19T01:16:44Zhttps://dspace.mit.edu/server/oai/request

oai:dspace.mit.edu:1721.1/1432472022-06-16T03:06:27Zcom_1721.1_7582com_1721.1_7581col_1721.1_131022

Broderick, Tamara Stephenson, William T. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science 2022-06-15T13:07:06Z 2022-06-15T13:07:06Z 2022-02 2022-03-04T20:47:50.655Z https://hdl.handle.net/1721.1/143247 Machine learning and statistical methods are increasingly used in high-stakes applications – for instance, in policing crime, making predictions about the atmosphere, or providing medical care. We want to assess the extent to which we can trust our methods, though, before we use them in such applications. There exist assessment tools, such as cross-validation (CV) and robustness checks, that help us understand exactly how trustworthy our methods are. In both cases (CV and robustness checks), a typical workflow follows the pattern of “change the dataset or method, and then rerun the analysis.” However, this workflow (1) requires users to specify the set of relevant changes, and (2) requires a computer to repeatedly refit the model. For methods involving large and complex models, (1) is expensive in terms of user time, and (2) is expensive in terms of compute time. So CV, which requires (2), and robustness checks, which often require both (1) and (2), see little use in the large and complex models that need them the most. In this thesis, we address these challenges by developing model evaluation tools that are fast in terms of both compute and user time. We develop tools to approximate CV when it is most computationally expensive: in high dimensional and complex, structured models. But approximating CV implicitly relies on the quality of CV itself. We show theory and empirics calling into question the reliability of the use of CV for quickly and automatically tuning model hyperparameters – even in cases where the behavior of CV is thought to be relatively well-understood. On the front of robustness checks, we note that a common workflow in Bayesian prior robustness requires users to manually specify a set of alternative reasonable priors, a task that can be time consuming and difficult. We develop automatic tools to search for a prediction-changing alternative prior for Gaussian processes, saving users from having to manually specify the set of alternative priors. Ph.D. Massachusetts Institute of Technology In Copyright - Educational Use Permitted Copyright MIT http://rightsstatements.org/page/InC-EDU/1.0/ Faster and easier: cross-validation and model robustness checks Thesis application/pdf Doctoral Doctor of Philosophy Publication unknown unknown <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="da583426-6424-4ce3-8121-598f28a96ed0"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Title>Faster and easier: cross-validation and model robustness checks</Title> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2022-02</PublicationDate> <Authors> <Author> <DisplayName>Stephenson, William T.</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <DisplayName>Massachusetts Institute of Technology</DisplayName> <OrgUnit /> </Publisher> </Publishers> <License>http://rightsstatements.org/page/InC-EDU/1.0/</License> <Abstract>Machine learning and statistical methods are increasingly used in high-stakes applications – for instance, in policing crime, making predictions about the atmosphere, or providing medical care. We want to assess the extent to which we can trust our methods, though, before we use them in such applications. There exist assessment tools, such as cross-validation (CV) and robustness checks, that help us understand exactly how trustworthy our methods are. In both cases (CV and robustness checks), a typical workflow follows the pattern of “change the dataset or method, and then rerun the analysis.” However, this workflow (1) requires users to specify the set of relevant changes, and (2) requires a computer to repeatedly refit the model. For methods involving large and complex models, (1) is expensive in terms of user time, and (2) is expensive in terms of compute time. So CV, which requires (2), and robustness checks, which often require both (1) and (2), see little use in the large and complex models that need them the most. In this thesis, we address these challenges by developing model evaluation tools that are fast in terms of both compute and user time. We develop tools to approximate CV when it is most computationally expensive: in high dimensional and complex, structured models. But approximating CV implicitly relies on the quality of CV itself. We show theory and empirics calling into question the reliability of the use of CV for quickly and automatically tuning model hyperparameters – even in cases where the behavior of CV is thought to be relatively well-understood. On the front of robustness checks, we note that a common workflow in Bayesian prior robustness requires users to manually specify a set of alternative reasonable priors, a task that can be time consuming and difficult. We develop automatic tools to search for a prediction-changing alternative prior for Gaussian processes, saving users from having to manually specify the set of alternative priors.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>