2026-09-19T01:16:26Zhttps://dspace.mit.edu/server/oai/request

oai:dspace.mit.edu:1721.1/1206602026-06-17T14:43:35Zcom_1721.1_7582com_1721.1_7581col_1721.1_131022

Jonathan Kelner and Tamara Broderick. Elder, Samuel Scott Massachusetts Institute of Technology. Department of Mathematics. Massachusetts Institute of Technology. Department of Mathematics 2019-03-01T19:56:01Z 2019-03-01T19:56:01Z 2018 2018 http://hdl.handle.net/1721.1/120660 1088419995 Thesis: Ph. D., Massachusetts Institute of Technology, Department of Mathematics, 2018. Cataloged from PDF version of thesis. Includes bibliographical references (pages 107-109). Validation refers to the challenge of assessing how well a learning algorithm performs after it has been trained on a given data set. It forms an important step in machine learning, as such assessments are then used to compare and choose between algorithms and provide reasonable approximations of their accuracy. In this thesis, we provide new approaches for addressing two common problems with validation. In the first half, we assume a simple validation framework, the holdout set, and address an important question of how many algorithms can be accurately assessed using the same holdout set, in the particular case where these algorithms are chosen adaptively. We do so by first critiquing the initial approaches to building a theory of adaptivity, then offering an alternative approach and preliminary results within this approach, all geared towards characterizing the inherent challenge of adaptivity. In the second half, we address the validation framework itself. Most common practice does not just use a single holdout set, but averages results from several, a family of techniques known as cross-validation. In this work, we offer several new cross-validation techniques with the common theme of utilizing training sets of varying sizes. This culminates in hierarchical cross-validation, a meta-technique for using cross-validation to choose the best cross-validation method. by Samuel Scott Elder. Ph.D. 109 pages eng Massachusetts Institute of Technology MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission. http://dspace.mit.edu/handle/1721.1/7582 Mathematics. Reliable validation : new perspectives on adaptive data analysis and cross-validation New perspectives on adaptive data analysis and cross-validation Thesis application/pdf false Publication unknown unknown <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="7e7c4314-a5de-4c77-85ad-9f519f018711"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Language>eng</Language> <Title>Reliable validation : new perspectives on adaptive data analysis and cross-validation</Title> <Subtitle>New perspectives on adaptive data analysis and cross-validation</Subtitle> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2018</PublicationDate> <Authors> <Author> <DisplayName>Elder, Samuel Scott</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <DisplayName>Massachusetts Institute of Technology</DisplayName> <OrgUnit /> </Publisher> </Publishers> <License>http://dspace.mit.edu/handle/1721.1/7582</License> <Keyword>Mathematics.</Keyword> <Abstract>Validation refers to the challenge of assessing how well a learning algorithm performs after it has been trained on a given data set. It forms an important step in machine learning, as such assessments are then used to compare and choose between algorithms and provide reasonable approximations of their accuracy. In this thesis, we provide new approaches for addressing two common problems with validation. In the first half, we assume a simple validation framework, the holdout set, and address an important question of how many algorithms can be accurately assessed using the same holdout set, in the particular case where these algorithms are chosen adaptively. We do so by first critiquing the initial approaches to building a theory of adaptivity, then offering an alternative approach and preliminary results within this approach, all geared towards characterizing the inherent challenge of adaptivity. In the second half, we address the validation framework itself. Most common practice does not just use a single holdout set, but averages results from several, a family of techniques known as cross-validation. In this work, we offer several new cross-validation techniques with the common theme of utilizing training sets of varying sizes. This culminates in hierarchical cross-validation, a meta-technique for using cross-validation to choose the best cross-validation method.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>