The SpaseCroissant Oven: Automatic Metadata Generation For Open-Source Space Weather Datasets
Name
chen-edenna-meng-eecs-2025-thesis.pdf
Description
Thesis PDF
Size
1.89 MB
Format
Adobe PDF
Checksum (MD5)
20a05438959a83d2d215aa66c1b244cb
Author(s)
Chen, Edenna H.
Advisor(s)
Devadas, Srini
How, Jonathan P.
Date Issued
February 2025
Publisher
Massachusetts Institute of Technology
Abstract
The rise of machine learning (ML) algorithms has led to a parallel rise in ML-ready datasets. A novel metadata schema released by OpenAI and MLCommons called Croissant, which is specifically designed for ML-ready datasets, aims to increase data accessibility, user understanding of data, and accuracy of claims based on data. However, current methods to automatically generate Croissant metadata present difficulties, such as involving manual entries. This can be especially difficult when attempting to preserve information about large ML-ready datasets, which are often derived from large scientific repositories belonging to organizations such as National Aeronautics and Space Administration (NASA). These major scientific repositories provide their own metadata standards, such as NASA’s Space Physics Archive Search and Extract (SPASE) schema but context from this metadata can often be lost during data processing. This thesis presents a novel, improved approach to Croissant metadata generation which involves a hybrid parsing logic and Large Language Model (LLM) inference approach, as well as recommendations for future Croissant standards and SPASE to Croissant schema metadata conversion, that aims to retain this lost context.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
Copyright retained by author(s)
Persistent DSpace Link
