Wednesday, August 15, 2012

False Wisdom of Crowds(?)

False Wisdom of Crowds,” Thomas Frey's recent blog post, comes to a useful conclusion – that analytical skills are becoming increasingly important to take advantage of big data – but some of his comments regarding the “wisdom of crowds” are incomplete.

For example, he says, “When working with large groups of people online, the wisdom of crowds is neither elevated to the smartest among them, nor is it diminished to the lowest levels. It hoovers somewhere in the middle.”

In fact, this is true for groups of all sizes, as evidenced by a demonstration I often use. Here is a specific, recent result.

During a presentation I was giving, I asked the audience to estimate the population of Albany, New York (capital of the Empire State and my home town). Here's the process they followed.

  1. I asked them to form into groups of three persons each. To each group I gave four index cards. One of the cards was marked “group estimate.” I asked them to hold that card aside.
  2. They made independent, individual estimates, and wrote each estimate one of the three remaining cards.
  3. They talked and arrived at a group estimate (using whatever method they wanted) and wrote it on the "group estimate" card.
  4. I showed them the 2010 Census estimate of Albany's population. They calculated the difference between the Census figure and each estimate and wrote it on the corresponding card.
  5. They ordered the four cards, the one closest to the Census figure at the top.
  6. I asked them "What is the rank of your Group Estimate? Is it the best estimate, second best, third best?" Here are the results.

Number of groups whose group estimate was …

2nd Best:6
3rd Best:4
4th Best:0

What this means is that six of the groups had one group member whose independent, individual estimate was better than the one they came to as a group; in four of the groups, there were two such group members. In only one group was the group's estimate better than that of any of the individuals.

This result is typical. (If there is anything unusual about it, it is that one of the groups had a group estimate that was the best.)

So one could conclude that groups are not effective. That would be the wrong conclusion.

Instead, researchers have focused on the processes used by groups to make estimates, judgments, and decisions. Using carefully thought out and tested techniques, it is possible to have groups consistently produce results that are better than that produced by the best individual working alone.* Indeed, by sharing information and correcting each others errors, groups can produce results that are better than the best individual.

* For example, see Reagan-Cirincione, P. (1994). “Improving the Accuracy of Group Judgment: A Process Intervention Combining Group Facilitation, Social Judgment Analysis, and Information Technology.” Organizational Behavior and Human Decision Processes, Vol. 58, No. 2 (May), pp. 246-270.


  1. "groups of three persons each" is not a crowd, it's a committee.

    The aggregate off all the individual guesses in the room would have been crowd sourcing.

    Did I misunderstand?

  2. Thanks for your comment.

    My post was intended to be about group performance, and in particular to address Frey's comment, which implicitly compared group performance to individual performance. (Crowdsourcing, as I understand it, does not necessarily involve group performance. That is, contributions can be solicited and collected from individuals and each contribution can be evaluated on its own merit -- without any process that resulted in a group product.)

    While the example I gave used groups of three (a choice driven largely by the seating arrangements in the room) I have used the same demonstration with larger groups, including everyone in the room, with similar results.

    I hope I understood your comment and said something intelligent in response!

  3. Sandy, the paper you cite seems interesting but there are no tables or description of the methodology in the abstract. Could you shed any light on these?

    1. It sounds like you have the abstract but not the full article. Am I right about that? If so, you should be able to obtain the full article from your local library through interlibrary loan. In a separate blog post I will excerpt Table 1, an overview of the process intervention, although some of it is cryptic without the entire article.


Any comments?