Evaluating Prediction Mechanisms: A Profitability Test
Author(s)
Sethi, Rajiv; Seager, Julie; Cai, Emily; Benjamin, Daniel; Morstatter, Fred; Bobrownicki, Olivia; Cheng, Yuqi; Kumar, Anushka; Wanganoo, Anusha; ... Show more Show less
Download3643562.3672612.pdf (1.967Mb)
Publisher Policy
Publisher Policy
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Terms of use
Metadata
Show full item recordAbstract
Any forecasting model can be represented by a virtual trader in a prediction market, endowed with a budget, risk preferences, and beliefs inherited from the model. We propose and implement a profitability test for the evaluation of forecasting models based on this idea. The virtual trader enters a position and adjusts its portfolio over time in response to changes in the model forecast and market prices, and its profitability can be used as a measure of model accuracy. We implement this test using probabilistic forecasts for competitive states in the 2020 US presidential election and congressional elections in 2020 and 2022, using data from three sources: model-based forecasts published by The Economist and FiveThirtyEight, and prices from the PredictIt exchange. The proposed approach can be applied more generally to any forecasting activity as long as models and markets referencing the same events exist.
Description
CI ’24, June 27–28, 2024, Boston, MA, USA
Date issued
2024-06-27Department
Sloan School of ManagementPublisher
ACM|Collective Intelligence Conference
Citation
Sethi, Rajiv, Seager, Julie, Cai, Emily, Benjamin, Daniel, Morstatter, Fred et al. 2024. "Evaluating Prediction Mechanisms: A Profitability Test."
Version: Final published version
ISBN
979-8-4007-0554-0