Monday, May 28, 2007

Web 2.0 and scientific method

It looks like people are puzzled as to why scientists aren't keen on Web 2.0 technologies. Being an physicist by training, even if not entirely by profession these days, at least to me, it's really obvious. The scientific process isn't about the wisdom of crowds. Just because everyone thinks something is true, it doesn't mean it is true...

While peer review has been compared to the wisdom of crowds, it isn't the same thing at all. Under the peer review model all people are not created equal, and expert knowledge of the subject area is both necessary and respected by the process. An obvious counter analogy in the Web 2.0 world is Wikipedia, which seems to lack the traditionally respect for expertise, and has suffered a large amount of criticism as a result.

Science has been built off the back of the peer review process, and anything which threatens to undermine that has to be looked at very, very, carefully. Not because change is necessarily bad, as scientists we're open to change, willing (hopefully) to throw away long treasured theories if the evidence proves that we've been wrong (or at least not right). But right now, there doesn't seem to be a lot of evidence that the wisdom of crowds will work out better than peer review.

For instance several times I've had to explain tagging to my colleagues because of its relevance to the ADS archive or arXiv pre-print server, and been met with exclamations of horror. Instead of promoting interesting research people are afraid that the wisdom of crowds will flatten it, because something is "obviously wrong" valid research will get trampled down before it sees the light of day. Currently the way to bury someones research is to duplicate their experiment and prove them wrong, when it simply becomes a matter of a number of influential people tagging a paper as being dubious, so nobody bothers to read it again, then you have to start to worry.