A strategy for determining the correct answer to a question posed to a group, when the most popular or most confident response fails under the same circumstances, is described this week in Nature.
The consensus of a group is often considered to be the best answer to a particular question because this approach takes advantage of the ‘wisdom of the crowd’. However, this method does not always produce the correct response because the most popular or confident solution can mask the minority opinion of a few individuals who may have specialized knowledge of the topic. For example, when a group was asked whether or not Philadelphia is the capital of Pennsylvania, Drazen Prelec and colleagues show that the popular vote endorses the incorrect answer (‘yes’ - the capital is actually Harrisburg).
The authors propose an alternative to this democratic vote that instead asks respondents to predict, in addition to casting their own vote, the distribution of others’ answers to the same question. By selecting the answer that is more popular than people predict (in this case ‘no’) the authors demonstrate that they can reliably identify the correct answer. They test this ‘surprisingly popular’ algorithm across a variety of scenarios in experiments involving the US state capitals and general knowledge, as well as by asking professional dermatologists to classify skin lesion images and asking art professionals and laypeople to estimate the value of various pieces of artwork.
In these studies, which involved different groups of 20-50 participants, Prelec and colleagues show that their algorithm reduced errors by approximately 21-35% relative to other common selection principles, such as choosing the simple majority or weighting votes by confidence. They note that this technique would be useful for questions about controversial topics, such as politics or the environment, where respondents might alter their vote predictions to align with their own views.