Fast and accurate Bayesian optimization with pre-trained transformers for constrained engineering problems

Abstract

Bayesian Optimization (BO) is a foundational strategy in engineering design optimization for efficiently handling black-box functions with many constraints and expensive evaluations. This paper introduces a novel constraint-handling framework for Bayesian Optimization (BO) using Prior-data Fitted Networks (PFNs), a foundation transformer model. Unlike traditional approaches requiring separate Gaussian Process (GP) models for each constraint, our framework leverages PFN’s transformer architecture to evaluate objectives and constraints simultaneously in a single forward pass using in-context learning. Through comprehensive benchmarking across 15 test problems spanning synthetic, structural, and engineering design challenges, we demonstrate an order of magnitude speedup while maintaining or improving solution quality compared to conventional GP-based methods with constrained expected improvement (CEI). Our approach particularly excels at engineering problems by rapidly finding feasible, optimal solutions. This benchmark framework for evaluating new BO algorithms in engineering design will be published at https://github.com/rosenyu304/BOEngineeringBenchmark .