2026-09-19T14:43:10Zhttps://dspace.mit.edu/server/oai/request

oai:dspace.mit.edu:1721.1/1636842025-11-18T06:27:25Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023

Kraska, Tim Russo, Matthew D. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science 2025-11-17T19:07:08Z 2025-11-17T19:07:08Z 2025-05 2025-08-14T19:33:16.534Z https://hdl.handle.net/1721.1/163684 Recently, AI developers have turned to modular AI systems in order to achieve state-ofthe-art performance on challenging benchmarks and industry problems. New programming frameworks have enabled developers to build these systems by composing them out of semantic operators—i.e., LLM-powered maps, filters, joins, aggregations, etc.—inspired by relational operators from data management systems. While these systems of semantic operators can achieve strong performance on benchmarks, they can be difficult to optimize. For example, an optimizer may need to determine which model, prompting strategy, and retrieval mechanism to use for each operator. Existing optimizers are limited in the number of optimizations they can apply, and most (if not all) cannot optimize system quality, cost, or latency subject to constraint(s) on the other dimensions. In this thesis, we build an extensible, cost-based optimizer called Abacus, which searches for the best implementation of a semantic operator system given a (possibly constrained) optimization objective. The optimizer estimates operator performance by leveraging a minimal set of training examples and, if available, prior beliefs about operator performance. We evaluate the optimizer on a range of workloads including biomedical multi-label classification (BioDEX), information extraction from legal contracts (CUAD), and multi-modal question answering (MMQA). We demonstrate that systems optimized by our work achieve 18.7%-39.2% better quality and up to 23.6x lower cost and 4.2x lower latency than the next best system. S.M. Massachusetts Institute of Technology In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ Cost-Based Optimization for Semantic Operator Systems Thesis application/pdf Master Master of Science in Electrical Engineering and Computer Science Publication unknown unknown <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="8c083748-b9b4-423b-a8b1-e10d68a956cd"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Title>Cost-Based Optimization for Semantic Operator Systems</Title> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2025-05</PublicationDate> <Authors> <Author> <DisplayName>Russo, Matthew D.</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <DisplayName>Massachusetts Institute of Technology</DisplayName> <OrgUnit /> </Publisher> </Publishers> <License>https://rightsstatements.org/page/InC-EDU/1.0/</License> <Abstract>Recently, AI developers have turned to modular AI systems in order to achieve state-ofthe-art performance on challenging benchmarks and industry problems. New programming frameworks have enabled developers to build these systems by composing them out of semantic operators—i.e., LLM-powered maps, filters, joins, aggregations, etc.—inspired by relational operators from data management systems. While these systems of semantic operators can achieve strong performance on benchmarks, they can be difficult to optimize. For example, an optimizer may need to determine which model, prompting strategy, and retrieval mechanism to use for each operator. Existing optimizers are limited in the number of optimizations they can apply, and most (if not all) cannot optimize system quality, cost, or latency subject to constraint(s) on the other dimensions. In this thesis, we build an extensible, cost-based optimizer called Abacus, which searches for the best implementation of a semantic operator system given a (possibly constrained) optimization objective. The optimizer estimates operator performance by leveraging a minimal set of training examples and, if available, prior beliefs about operator performance. We evaluate the optimizer on a range of workloads including biomedical multi-label classification (BioDEX), information extraction from legal contracts (CUAD), and multi-modal question answering (MMQA). We demonstrate that systems optimized by our work achieve 18.7%-39.2% better quality and up to 23.6x lower cost and 4.2x lower latency than the next best system.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>