Experimental Design in Operations

Abstract

Experimental design has been fundamental to many fields, yet applications in operations research (OR) and operations management (OM) bring in complexities, as well as opportunities, that extend beyond classical statistical goals. This thesis discusses why OR/OM should care about experimentation and its design, where the challenges lie in operational and service systems for classical experimental design, and why OR/OM researchers are uniquely suited to address the challenges. More specifically, this thesis advances experimental design by introducing more operational perspectives, addressing two core challenges: incorporating operational objectives and leveraging operational modeling to enhance experimentation.

First, traditional experimental approaches, such as A/B testing, primarily aim at statistical efficiency (e.g., reducing variance or bias). However, OR/OM applications frequently involve additional operational considerations, such as welfare preservation, revenue optimization, risk control, and non-stationarity. We investigate these settings in Chapters 2–4, developing frameworks for multi-objective experimental design. In Chapter 2, we introduce a minimax multi-objective optimization formulation to balance statistical power and welfare loss, derive necessary and sufficient conditions for Pareto optimal solutions, and propose robust multi-armed bandit designs. Chapter 3 extends this approach to pricing experiments, exploring trade-offs between estimating causal effects (price elasticity), maximizing revenue, and controlling tail risks, along with robust statistical inference methods. Chapter 4 addresses non-stationary experimental environments where treatment effects dynamically evolve, designing experiments that optimally balance accurate estimation of changing effects and welfare minimization.

Furthermore, we highlight the substantial value of operational models—particularly Markov Decision Processes (MDPs)—in experimental design. In Chapter 5, we address the challenge of estimating long-term cumulative outcomes, such as customer lifetime value, using short-term experimental data. We develop novel inference methods grounded in MDPs, which effectively bridge short-term data to long-term outcomes. Moreover, by recognizing many real-world treatments tend to be localized for practical efficiency, we introduce novel estimators that leverage the localized structures to achieve substantial variance reductions.

In summary, this thesis underscores how OR/OM contexts uniquely enrich experimental design, offering robust theoretical frameworks and practical solutions to operational challenges, ultimately broadening both the theoretical foundations and the practical impacts of experimentation.