Deep Learning Multimodal Extraction of Reaction Data
Name
wang-wang7776-meng-eecs-2024-thesis.pdf
Description
Thesis PDF
Size
1.04 MB
Format
Adobe PDF
Checksum (MD5)
47e3a2b40232b390a526e889ee815d62
Author(s)
Wang, Alex
Advisor(s)
Barzilay, Regina
Date Issued
September 2024
Publisher
Massachusetts Institute of Technology
Abstract
Automated extraction of structured information from chemistry literature is vital for maintaining up-to-date databases for use in data-driven chemistry. However, comprehensive extractions require reasoning across multiple modalities and the flexibility to generalize across different styles of articles. Our work on OpenChemIE presents a multimodal system that reasons across text, tables, and figures to parse reaction data. In particular, our system is able to infer structures in substrate scope diagrams as well as align reactions with their metadata defined elsewhere. In addition, we explore the chemistry information extraction potential of Vision Language Models (VLM), which allow powerful large language models to leverage visual understanding. Our findings indicate that VLMs still require additional work in order to meet the performance of our bespoke models.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
