How does meta-analysis work?

Question

I understand that meta-analysis combines the results of several inconclusive PK studies to get a statistically significant result. How does that work? The Skeptic's dictionary didn't help me much. Is it as simple as just choosing several small positive results to get greater odds against it being due to chance? If so, why would anyone accept this? It's just a higher order of counting the hits and ignoring the misses.

Answer 1

I think you're referring to the work that PEAR did. I've read their paper on this meta-analysis and they report that other studies reported statistically significant results, though I haven't obtained copies of those papers yet to see if it makes sense.

Meta-analysis is certainly fraught with pitfalls like the ones you mentioned, plus many more. It is not used in the physical sciences much apart from medical or health studies. It is a legitimate research method, but one that can be very misleading if it is not done with extreme care.

Regarding the PEAR group, they did use this method but they had to use parapsychology studies not from scientific peer-reviewed journals but from paranormal organization sources, and even then they only reported less than 0.05% deviation from the expected random distribution. When looking at this work, it is illuminating to recall Prof. Langmuir's 6 Characteristics of  Pathological (bad!) Science (see link):

1. The maximum effect that is observed is produced by a causative agent of barely detectable intensity, and the magnitude of the effect is substantially independent of the intensity of the cause.

2. The effect is of a magnitude that remains close to the limit of detectability; or, many measurements are necessary because of the very low statistical significance of the results.

3. Claims of great accuracy.

4. Fantastic theories contrary to experience.

5. Criticisms are met by ad hoc excuses thought up on the spur of the moment.

6. Ratio of supporters to critics rises up to somewhere near 50% and then falls gradually to oblivion.

Most of these seem to apply to PEAR, most significantly #1 and #2. I cannot confirm #5 and as far as #6 they never had anywhere close to that 50% ratio.

As far as I know, meta-analysis is not used at all in the physics literature. I know there is at least one physics PhD candidate here who might be able to confirm that.