MIT Libraries logoDSpace@MIT

MIT
View Item 
  • DSpace@MIT Home
  • MIT Libraries
  • MIT Theses
  • Graduate Theses
  • View Item
  • DSpace@MIT Home
  • MIT Libraries
  • MIT Theses
  • Graduate Theses
  • View Item
JavaScript is disabled for your browser. Some features of this site may not work without it.

Limiting bias from test-control interference in online marketplace experiments

Author(s)
Holtz, David Michael
Thumbnail
DownloadFull printable version (3.344Mb)
Other Contributors
Sloan School of Management.
Advisor
Sinan Aral.
Terms of use
MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission. http://dspace.mit.edu/handle/1721.1/7582
Metadata
Show full item record
Abstract
Many internet firms use A/B tests to make product decisions. When running an A/B test, the typical objective is to measure the average treatment effect (ATE), i.e. the difference between the average outcome in the counterfactual situation where 100% of users are exposed to the treatment and the average outcome in the status quo situation (in which 100% of users are exposed to the control). However, a simple difference-in-means estimator will give a biased estimate of the ATE when outcomes of units in the control depend on the behavior or outcomes of units in the treatment - a case referred to in this work as test-control interference. Previous work has found evidence of bias due to test-control interference in online marketplace experiments. Using a simulation built on top of scraped Airbnb data, this paper considers the use of experiment designs and ATE estimators from the network interference literature for online marketplace experimentation. I first model the marketplace as a network in which an edge exists between two sellers if their goods substitute for one another. I then create an agent-based simulation to model seller outcomes, both under the status quo and when subjected to "treatments" that force hosts to lower their price. I then use the same simulation framework to approximate ATE distributions obtained when various network experiment design and analysis techniques are used. I find that graph cluster randomization leads to bias reductions of as much as 62%. Unfortunately, the variance of ATE estimators also increases significantly. Replacing the simple difference-in-means estimator with more sophisticated ATE estimators can lead to mixed results. While some methods (i.e., exposure models) provide (small) additional reductions in bias and small reductions in variance, others (i.e., the Hajek estimator for the ATE) lead to increased bias and variance. Although further work is needed, current results suggest that experiment design and analysis techniques from the network experimentation literature are promising tools for for reducing bias due to test-control interference in marketplace experiments.
Description
Thesis: S.M. in Management Research, Massachusetts Institute of Technology, Sloan School of Management, 2018.
 
Cataloged from PDF version of thesis.
 
Includes bibliographical references (pages 37-38).
 
Date issued
2018
URI
http://hdl.handle.net/1721.1/117999
Department
Sloan School of Management
Publisher
Massachusetts Institute of Technology
Keywords
Sloan School of Management.

Collections
  • Graduate Theses

Browse

All of DSpaceCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

My Account

Login

Statistics

OA StatisticsStatistics by CountryStatistics by Department
MIT Libraries
PrivacyPermissionsAccessibilityContact us
MIT
Content created by the MIT Libraries, CC BY-NC unless otherwise noted. Notify us about copyright concerns.