Limiting bias from test-control interference in online marketplace experiments

Abstract

Many internet firms use A/B tests to make product decisions. When running an A/B test, the typical objective is to measure the average treatment effect (ATE), i.e. the difference between the average outcome in the counterfactual situation where 100% of users are exposed to the treatment and the average outcome in the status quo situation (in which 100% of users are exposed to the control). However, a simple difference-in-means estimator will give a biased estimate of the ATE when outcomes of units in the control depend on the behavior or outcomes of units in the treatment - a case referred to in this work as test-control interference. Previous work has found evidence of bias due to test-control interference in online marketplace experiments. Using a simulation built on top of scraped Airbnb data, this paper considers the use of experiment designs and ATE estimators from the network interference literature for online marketplace experimentation. I first model the marketplace as a network in which an edge exists between two sellers if their goods substitute for one another. I then create an agent-based simulation to model seller outcomes, both under the status quo and when subjected to "treatments" that force hosts to lower their price. I then use the same simulation framework to approximate ATE distributions obtained when various network experiment design and analysis techniques are used. I find that graph cluster randomization leads to bias reductions of as much as 62%. Unfortunately, the variance of ATE estimators also increases significantly. Replacing the simple difference-in-means estimator with more sophisticated ATE estimators can lead to mixed results. While some methods (i.e., exposure models) provide (small) additional reductions in bias and small reductions in variance, others (i.e., the Hajek estimator for the ATE) lead to increased bias and variance. Although further work is needed, current results suggest that experiment design and analysis techniques from the network experimentation literature are promising tools for for reducing bias due to test-control interference in marketplace experiments.