False Wisdom of Crowds(?)

“False Wisdom of Crowds,” Thomas Frey's recent blog post, comes to a useful conclusion – that analytical skills are becoming increasingly important to take advantage of big data – but some of his comments regarding the “wisdom of crowds” are incomplete.

For example, he says, “When working with large groups of people online, the wisdom of crowds is neither elevated to the smartest among them, nor is it diminished to the lowest levels. It hoovers somewhere in the middle.”

In fact, this is true for groups of all sizes, as evidenced by a demonstration I often use. Here is a specific, recent result.

During a presentation I was giving, I asked the audience to estimate the population of Albany, New York (capital of the Empire State and my home town). Here's the process they followed.

1. I asked them to form into groups of three persons each. To each group I gave four index cards. One of the cards was marked “group estimate.” I asked them to hold that card aside.
2. They made independent, individual estimates, and wrote each estimate one of the three remaining cards.
3. They talked and arrived at a group estimate (using whatever method they wanted) and wrote it on the "group estimate" card.
4. I showed them the 2010 Census estimate of Albany's population. They calculated the difference between the Census figure and each estimate and wrote it on the corresponding card.
5. They ordered the four cards, the one closest to the Census figure at the top.
6. I asked them "What is the rank of your Group Estimate? Is it the best estimate, second best, third best?" Here are the results.

Number of groups whose group estimate was …

Best:	1
2nd Best:	6
3rd Best:	4
4th Best:	0

What this means is that six of the groups had one group member whose independent, individual estimate was better than the one they came to as a group; in four of the groups, there were two such group members. In only one group was the group's estimate better than that of any of the individuals.

This result is typical. (If there is anything unusual about it, it is that one of the groups had a group estimate that was the best.)

So one could conclude that groups are not effective. That would be the wrong conclusion.

Instead, researchers have focused on the processes used by groups to make estimates, judgments, and decisions. Using carefully thought out and tested techniques, it is possible to have groups consistently produce results that are better than that produced by the best individual working alone.* Indeed, by sharing information and correcting each others errors, groups can produce results that are better than the best individual.

* For example, see Reagan-Cirincione, P. (1994). “Improving the Accuracy of Group Judgment: A Process Intervention Combining Group Facilitation, Social Judgment Analysis, and Information Technology.” Organizational Behavior and Human Decision Processes, Vol. 58, No. 2 (May), pp. 246-270.