2026-09-19T00:58:22Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1627422025-12-09T18:27:23Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Stonebraker, Michael R.
Choi, Justin J.
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2025-09-18T14:30:08Z
2025-09-18T14:30:08Z
2025-05
2025-06-23T14:01:39.177Z
https://hdl.handle.net/1721.1/162742
This work examines the current state of using large language models (LLMs) to solve Text-to-SQL tasks on databases in an enterprise setting. Benchmarks on publicly available datasets do not fully capture the difficulty and complexity of this task in a real-world, enterprise setting. This study examines the critical steps needed to work with enterprise data as well as using knowledge-injection to enhance the performance of LLMs on Text-to-SQL tasks. We begin by evaluating the baseline performance of LLMs on enterprise databases, revealing that a predominant source of failure stems from a lack of domain-specific knowledge. To improve performance, we explore knowledge-injection: the process of incorporating internal and external knowledge. Internal knowledge consists of database-specific information such as join logic, while external knowledge refers to institutional acronyms or group names. We present a hybrid retrieval pipeline that combines embedding and text based searching with LLM-guided ranking to supply models with relevant external knowledge during Text-to-SQL generation. We evaluate the impact of the knowledge-injection by testing the performance of LLMs on the table retrieval task after being augmented with appropriate external knowledge. We demonstrate that knowledge-injection significantly improves accuracy on table retrieval using BEAVER: an enterprise-level Text-to-SQL benchmark. Our findings highlight the importance of domain-specific knowledge-injection and retrieval augmentation in bringing LLMs closer to deployment in enterprise-grade database systems, as well as common failure modes that occur when executing enterprise Text-to-SQL.
M.Eng.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright retained by author(s)
https://rightsstatements.org/page/InC-EDU/1.0/
Injection of Domain-Specific Knowledge for Enterprise Text-to-SQL
Thesis
application/pdf
Master
Master of Engineering in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="97d3c056-ad9d-4591-ae12-75053a6d688a">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>Injection of Domain-Specific Knowledge for Enterprise Text-to-SQL</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2025-05</PublicationDate>
<Authors>
<Author>
<DisplayName>Choi, Justin J.</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>https://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>This work examines the current state of using large language models (LLMs) to solve Text-to-SQL tasks on databases in an enterprise setting. Benchmarks on publicly available datasets do not fully capture the difficulty and complexity of this task in a real-world, enterprise setting. This study examines the critical steps needed to work with enterprise data as well as using knowledge-injection to enhance the performance of LLMs on Text-to-SQL tasks. We begin by evaluating the baseline performance of LLMs on enterprise databases, revealing that a predominant source of failure stems from a lack of domain-specific knowledge. To improve performance, we explore knowledge-injection: the process of incorporating internal and external knowledge. Internal knowledge consists of database-specific information such as join logic, while external knowledge refers to institutional acronyms or group names. We present a hybrid retrieval pipeline that combines embedding and text based searching with LLM-guided ranking to supply models with relevant external knowledge during Text-to-SQL generation. We evaluate the impact of the knowledge-injection by testing the performance of LLMs on the table retrieval task after being augmented with appropriate external knowledge. We demonstrate that knowledge-injection significantly improves accuracy on table retrieval using BEAVER: an enterprise-level Text-to-SQL benchmark. Our findings highlight the importance of domain-specific knowledge-injection and retrieval augmentation in bringing LLMs closer to deployment in enterprise-grade database systems, as well as common failure modes that occur when executing enterprise Text-to-SQL.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>