Measuring the effects of online advertising on human behavior using natural and field experiments

Abstract

This thesis investigates the effects of online advertising on human behavior: clicks, new-account sign-ups, and retail sales. Five chapters cover natural and field experiments used to measure these effects for both display and search advertising. The first chapter uses a natural experiment on the Yahoo! Front Page, aided by a flexible semiparametric model, to identify the causal effects of display ad frequency on internet users' responses as measured at the individual level by clicks and new-account sign-ups. Performance is heterogeneous regarding frequency and clickability; some campaigns exhibit significant decreasing returns to scale after one or two impressions while others show constant returns to scale even after fifty impressions. For some campaigns, a simple nonparametric regression which ignores selection bias finds increasing returns to scale, but none is found with the model that uses exogenous variation in views. Conversely, many campaigns that appear to exhibit diminishing returns when failing to account for selection, in fact, show little to no wear-out. The second chapter assesses the ability of online display advertising to attract new customers by analyzing a large-scale field experiment which exposed 3.7 million subjects to ads on Yahoo!. The number of new account sign-ups at an online business was tracked and shows a statistically significant impact of one of the two types of advertising campaigns. The ads served as Yahoo! run-of-network succeeded in generating a statistically significant increase in sign-ups of 8-14% relative to the control group. The ads shown on Yahoo! Mail did not produce a statistically significant increase in sign-ups. Despite being derived using millions of subjects, this estimate is quite noisy, with the upper bound of the 95% confidence interval estimate being a 15% increase in new customers. These estimates call into question click-only attribution models, as the number of users that clicked on an ad and converted is less than 30% of the estimated treatment effect. The third chapter asks, "Does advertising affect sales in a measurable way?" New technologies for tracking both sales and advertising at the individual level are used to investigate the effectiveness of brand advertising for a nationwide retailer. A controlled experiment on 1,577,256 existing customers measures the causal effect of advertising on actual purchases, overcoming the major hurdles regarding attribution typically encountered in advertising effectiveness research by exogenously varying exposure to the ads. Each customer was randomly assigned to treatment and control groups for an online advertising campaign for this retailer. Online brand advertising generated a statistically and economically significant effect on in-store sales for this retailer. The design of the experiment permits a demographic breakdown of the advertising's heterogeneous effects. Somewhat surprisingly, the effects are especially large for the elderly. Customers aged 65 and older, comprising only 5% of the experimental subjects, exhibited a 20% average increase in sales due to the advertising campaign, which represents 40% of the total effect among all age groups. The fourth chapter further investigates the effects of online advertising on sales. A quasi experimental approach is taken to analyze the randomized experiment in Chapter 3. Individual level data on ad exposure and weekly purchases at this retailer, both online and in stores, are combined and used to find statistically and economically significant impacts of the advertising on sales. The treatment effect persists for weeks after the end of an advertising campaign, and the total effect on revenues is estimated to be more than seven times the retailer's expenditure on advertising during the study. Additional results explore differences in the number of advertising impressions delivered to each individual, online and offline sales, and the effects of advertising on those who click the ads versus those who merely view them. The fifth chapter quantifies the externalities exerted among competing north ads in search advertising. "North" ads, or sponsored listings appearing just above the organic search results, generate the majority of clicks and revenues for search engines. The research question asks, "Does increasing the number of rival north ads decrease the number of clicks I receive on my own north ad?" A controlled experiment investigates this question and finds, surprisingly, that additional rival ads in the north tend to increase rather than decrease the click-through rate (CTR) of the top sponsored listing. Several possible explanations are proposed for this phenomenon, and directions for new theoretical models of sponsored search are suggested.