Identify experts through Revealed Confidence : application to Wisdom of Crowds

Abstract

We propose our Revealed Confidence (RC) algorithm that improves Wisdom of Crowds (WoC) by identifying experts from the crowds. We highlight the important distinction between first- and second-order uncertainty, which also serves as an explanation for rational overconfidence. Under our proposed belief updating mechanism, we analyze the performance of RC algorithm and show the algorithm could identify the more accurate prior estimates even if all agents report the same prior confidence under conventional confidence elicitation, e.g. confidence interval. Our empirical analysis shows that (1) RC improves upon other wisdom of Crowds methods by overweighting the more accurate agents in the aggregation (2) verifies one key prediction of our theoretical result that the distance effect indeed affects belief-updating henceforth RC algorithm's performance, which should be carefully controlled for in order to optimize the algorithm..