Constrained and High-dimensional Bayesian Optimization with Transformers
Name
yu-rosenyu-csesm-ccse-2025-thesis Rosen Yu.pdf
Description
Thesis PDF
Size
3.33 MB
Format
Adobe PDF
Checksum (MD5)
250dd6bf945ca67452c95724f03a8f04
Author(s)
Yu, Rosen Ting-Ying
Advisor(s)
Ahmed, Faez
Date Issued
May 2025
Publisher
Massachusetts Institute of Technology
Abstract
This thesis advances Bayesian Optimization (BO) methodology through two novel algorithms that address critical limitations in handling constraints and high-dimensional spaces. First, we introduce a constraint-handling framework leveraging Prior-data Fitted Networks (PFNs), a foundation transformer model that evaluates objectives and constraints simultaneously in a single forward pass through in-context learning. This approach demonstrates an order of magnitude speedup while maintaining or improving solution quality across 15 test problems spanning synthetic, structural, and engineering design challenges. Second, we propose Gradient-Informed Bayesian Optimization using Tabular Foundation Models (GITBO), which utilizes pre-trained tabular foundation models as surrogates for high-dimensional optimization (exceeding 100 dimensions). By exploiting internal gradient computations to identify sensitive optimization directions, GIT-BO creates continuously re-estimated active subspaces without model retraining. Empirical evaluation across 23 benchmarks demonstrates GIT-BO’s superior performance compared to state-of-the-art Gaussian Process-based methods, particularly as dimensionality increases to 500 dimensions. Together, these approaches establish foundation models as powerful alternatives to Gaussian Process methods for constrained and high-dimensional Bayesian optimization challenges.
MIT Department
Massachusetts Institute of Technology. Center for Computational Science and Engineering
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
