What does peer review mean to you?

Question

I believe that in today’s politically charged world that peer review means nothing more than a bunch of like-minded scientists agreeing with each other.

Mann’s hockey stick was peer reviewed. It was published in Nature Magazine and used as the cornerstone to the IPCC 3rd assessment. Nobody questioned the methodology or the data, which were both flawed. Nobody checked Mann’s work. They all just accepted it as gospel because it fit neatly into their preconceived idea of man-made global warming.

It took mathematician Stephen McIntyre and economist Ross McKitrick to identify Mann’s flaws. They wrote an article about their findings and submitted it to Nature – which of course rejected it. It took him five years to convince the National Academy of Sciences to review Mann’s work.

How many other supposedly peer reviewed papers fall into this category?

Answer 1

Exactly - Peer review is just buddies commenting on others work.  It's a very low standard first used by magazines to cover their buts.

Imagine people who formed a science based on UFOs being real.  Would their peer reviewed material prove UFOs were real?

However this low standard is just what the believers want to hear.

Answer 2

M&M's work did appear in a peer-reviewed journal, put out by AGU in 2005.  So peer-review does work in general.  Nature is super picky about the things they publish.  For example, my field of work will never appear in Nature becuase it will never be deemed "hot button" enough.  You can't really assess the entire world of peer-review based solely on how Nature chooses what to publish and not publish.

I encourage you to look up this peer-reviewed AMS journal: Rutherford, S., Mann, M.E., Osborn, T.J., Bradley, R.S., Briffa, K.R., Hughes, M.K., Jones, P.D., Proxy-based Northern Hemisphere Surface Temperature Reconstructions: Sensitivity to Methodology, Predictor Network, Target Season and Target Domain, Journal of Climate, in press (2005).  They do work to discredit McIntrye and McKitrick.

Here is Michael Mann's own personal rebuttal to M&M's paper which you would find interesting:

http://www.realclimate.org/index.php/arc...

I will add that peer-review is not perfect (it didn't work when that Korean scientist faked data a few years ago in Science), there are certainly issues with it, but it is better than having no work checked by others at all.  Otherwise I could publish a paper saying the moon were made of cheese and you all would have to take it as credible science.

Answer 3

"I believe that in today’s politically charged world that peer review means nothing more than a bunch of like-minded scientists agreeing with each other."

Unfortunately, such statements only come from non-scientists with very little understanding of the peer review process. Peer review is the best standard to date for quality assurance in scientific research and findings. Is it perfect? Not even close, but like some say, "it is a necessary, but not sufficient condition".

"How many other supposedly peer reviewed papers fall into this category?"

There are numerous examples of incorrect, fraudulent, or downright stupid work in the peer reviewed literature, but this does not mean we can write off the entire process as useless.

-------

Edit:

"However, their very existence demonstrates that a peer-reviewed paper is not unassailable."

This is a strawman. I don't think anyone has argued peer review as infallible.

fijijenn wrote

"For example, my field of work will never appear in Nature becuase it will never be deemed "hot button" enough."

If you don't mind me asking, what is your field of work, fijijenn?

Answer 4

Peer review means experts are verifying that the analysis in a scientific paper is valid.

By the way, McIntryre and McKitrick's critique was flawed.

"Some of you might be thinking, wouldn’t MM [McIntyre and McKitrick] have noticed that their fit (the one lacking a hockey stick) failed statistical significance when they did the verification step to check the correctness of their version? Probably yes. Too bad they simply didn’t do the verification step. But independent researchers have replicated the MM procedure and subjected it to verification. It failed."

http://tamino.wordpress.com/2008/03/06/p...

Which is why the NAS confirmed the basic accuracy of Mann's study.

I know it's hard to swallow, but the tens of thousands of climate scientists who agree with the consensus on AGW aren't part of some massive conspiracy.  Like most conspiracy theories, this one has no basis in reality.

Answer 5

Unfortunately, 'peer review' often means that a clique of folks with a similar agenda get to back-scratch each other and incestuously promote each other.

I prefer 'TRUTH' over 'peer-reviewed' any day.

Answer 6

Most scientific issues, it's esoteric, there are five or six different groups studying slightly different discreet issues and coming at a discipline from slightly different angles.    There is competition among them.   Peer review means one scientist who thinks X about issue Y reviews a paper within his discipline by a scientist who thinks A about issue B - there may be some overlap or disagreement, and it's precisely that competitive aspect that makes it meaningful that the second scientist vouches for the first.

With global warming there is a popular agenda, and research is funded by government, which has a stake in the outcome - - man-made global warming is the justification for a massive government takeover of the economy.    All of the science is about one specific question - does mankind contribute materially to the climate shift that occurred in the 20th century?    And the money being spent to research it isn't being spent to research the issue isn't spent to fund an open-ended search, but specifically to prove a specific answer to that specific question.    There's not a 99-1 consensus or even 90-10 but it's 80-20 or 75-25.    "Peer review" loses, then, its competitive nature.   It's almost like a Trust.    We have Bradley and Mann using the same data, the same methodologies, and having the same agenda, and - shocker of shockers, reaching the same conclusion - and then reviewing each other's work!    In the present context, then, "peer review" is meaningless, despite the degrees these men hold.   It's like having Alberto Gonzalez review the legality of Bush's spying.

As for the Hockey Stick - the day anyone explains a single example of the physical evidence of warmer temperturatures during the MWP, I will reconsider.  There are countless examples of things that happened - what grew where, etc... - that have since they happened been explained as resulting from warmer temperatures, and that do not happen today because it's not warm enough.  Again - if you're going to tell me it's scientifically impossible for Washington and his men to have crossed the Delaware, I'll consider believing you when you explain to me how he appeared the next morning on the other side.

People dismiss the Viking and British examples as "limited to Northern Europe."

Lake Naivasha, Kenya, dried up for 200 years.    That's not in Northern Europe.    The tree lines were higher in the Sierra Nevadas.    That's not in Northern Europe.   The droughts were much more severe in the American Southwest than they have been in the 20th century.    That's not in Northern Europe.

Answer 7

Obviously, you've never been through the process. Journals use peer review in attempt to catch errors before publication. And it does a pretty good job of that. It's not unusual for a submitted paper to garner dozens of critical comments during the peer review process, many of which may result in, or at times require, revision to the original paper. In fact, a paper that goes through peer-review without any modifications prior to publication is the exception rather than the rule.

Mann's hockey stick paper passed peer review because its conclusions were sound (and still are). The errors found by MacIntyre and McKitrick were small, and even if corrected would not have changed the paper's conclusions. The reason Nature did not publish M&M's criticism is because M&M's criticism itself failed their peer review. Although they did make it into another peer-reviewed publication.

I've read the original paper, and M&M's criticism, and Mann's rebuttal, and all I can say is that M&M's criticism is a very thin statistical beef. Mann's rebuttal is utterly convincing: the entire statistical argument can be avoided by dumping PCA statistics entirely and using all of the (thousands of) original data -- and when you do that, Mann's original conclusion is still fully supported, while M&M's conclusions are not.

Since Mann's paper was first published, about 20 other papers by other authors have re-analyzed the same data and come to the same conclusions. Here's a summary:

http://farm4.static.flickr.com/3067/2616...

Can you tell which one of those lines belongs to Mann 1999?