Why is meta-analysis condemned when used in parapsychology, yet it is accepted in other scientific research?

Question

Interesting, that my latest comment ended up there twice....

Answer 1

When all the individual analyses are already compromised due to poor experimental techniques, a meta-analysis cannot be any better.

I'll take one rigorously-controlled, double-blind parapsychology experiment over a self-applied meta-analysis any day.

Answer 2

Meta-analysis is a useful tool in all disciplines, but you have to make sure that all the studies you apply it to were done under rigorous conditions, and that typically is not the case in parapsychology - the more rigorous the standards, the less of a result you see.  They get the best results when they relax their standards and allow for their subjects to cheat.  Thus, many of these studies aren't up to the standards needed to apply a useful meta-analysis - they would skew the results.  Even if the meta-analysis was done correctly, the results aren't valid if any of the data points aren't valid.

Answer 3

You are correct meta-analysis (link below for those who don't know what this is) is used extensively in both social sciences and medical research.

Despite the claims often posted by skeptics (they allow their subjects to cheat) that are never supported by any evidence all claims in science (they cheat) require supporting evidence and until this is shown the hypothesis remains unsupported and the results of any questioned experiment that do provide supporting evidence stand.

For the record psychical researchers and parapsychologist have an excellent record of exposing frauds (and providing evidence) and openly and instantly disclose these incidents.

The supporting evidence for psi is in direct contradiction to skeptical claims (insignificant effects in single studies and poor methodology). In fact psi experiments frequently and with many replications produce effect sizes larger than those shown in medical studies (aspirin and anti-depressants for example). Meta-analysis is a  statistical technique used to find these small effects in medicine so that people would not be denied an effective (if not extraordinary) medicine. However, since many people are more politically motivated to stop psi research than medical research the same level of criticism is not applied to medical research. Thus, extraordinary results are expected ( a concept not based in science) in each and every experiment to show psi and the standards of evidence in science are conventionally ignored in support of a belief.

Regarding methods and controlled experiments. While many claims are leveled (usually falsely or misleadingly) against psi methods by comparison to other social and medical research psi methods are consistently better. (see the Automated Ganzfeld protocols).

This charge stems from most skeptics being unfamiliar with any actual research or psi experiments and often being unfamiliar with acceptable methods and evaluation techniques in both medicine and social research.

In fairness some of this may stem from the physical sciences where they are not dealing with human abilities and performance issues and are not familiar with research involving human or animal subjects. Basically, people with no education or training in the area of science they choose to comment (criticize) on.

Any even casual look through the psi scientific research literature (peer reviewed scientific journals) will support the answer above.

Several supporting links below

Psiexploration

Answer 4

Meta-analysis is used differently in different sciences.  

In the phsycial sciences, meta-analysis is typically used to combine partial results from one study with the partial results of another study.  For example, a study may be done that has many dependent variables.  One of those variables may be interesting to a researcher performing a meta-analysis.  The researcher will take the results of each study and try to isolate the effects of the variable that they are interested in studying.  This type of meta-analysis is typically done when it is not possible or feasible to create a single study to specifically test the variable in question.

Meta-analysis in the physical sciences is typically accepted more easily because of it's quantitative nature.  In other words, if numbers can be used to provide a proof, it's easier for most people to believe those proofs.

Within social sciences (sociology, psychology, parapsychology, etc.), meta-analysis is typically used to combine the results of smaller experiments to provide a sample size that is large enough to eliminate statistical anomalies.  Social science experiments typically do not include tens of thousands of trials done using the same protocol, and so, the studies must be combined to get a sample that is large enough to overcome normal statistical variations.  There are also qualitative aspects to social science experiments that cannot easily be evaluated using numbers - hence the disparaging moniker, "soft-science".

One of the criticisms of this approach to analysis is related to the fact that different methods are used to gather the data, and so, the data cannot be combined and treated as a single group of data.  This criticism would be valid if the data were being treated like it all came from a single study, but, in good meta-analysis, the data is rated before it is combined.  That is, data that is more directly related to the research work will be "weighted" more heavily than data that was only peripherally obtained.  This provides an even playing field for all of the information gathered before the data is combined.

Another criticism is that only the most successful data is being reported, and failed trials are being left "in the file drawer".  Again, this can be accounted for statistically, and the meta-analysis can provide for this variable.  

In this forum, meta-analysis is criticized related to parapsychology, but the same arguments used here could just as easily be used in other social science experiments.  The missing fact is that when meta-analysis is done correctly, the objections that I mentioned above can be overcome, and there is no reason to dismiss meta-analysis in any of the social sciences, including parapsychology.  As experimental techniques advance, and as analysis techniques are refined, the evidence will become more clear, and the objections to the techniques that are used will diminish.  Thanks for the question!

Answer 5

Simply put, the reason is because in other scientific disciplines meta-analysis is generally used for generating hypotheses for further testing. e.g., clinical testing of drug efficacy is one area. However, parapsychology stands unique from the rest of social, physical and medical science in the fact that it rests almost entirely on meta-analysis for support, which is akin to building your house on flimsy stilts instead of a concrete foundation. The danger in meta-analysis is that it can take a large number of poor quality, statistically insignificant results and put them together into what is supposed to be one large, statistically significant study, but many poor quality studies don't add up to be one good study. In fact, independent analysis of parapsychological meta-analyses have turned up a number of such problems. There's much more about meta-analysis in parapsychology in the link below.

EDIT: I do understand there is differing opinion about the quality of parapsychological research to date. I don't base my stated opinion on ignorance but on a careful study of the subject, and neither are my opinions uncommon among other scientists who have done the same, according to the reference I do provide. So it would be overly simplistic to assume that my position on this must be due to misunderstanding or lack of knowledge, and it would be no less valid if I were to claim that opinions contrary to mine are based on mistaken beliefs. Perhaps it's best to attribute it to difference in opinion and leave it at that. Regardless, I appreciate that this question was asked.

Answer 6

Skeptics apply illegitimate double standards when evaluating controversial vs. conventional topics. If the same standards of evaluating empirical evidence were used across disciplines, there would be no question that psi effects would be accepted as empirical fact. Issues about difficulties in strictly replicating effects, variations in experimental quality, combining "apples" and "oranges" in meta-analysis, the selective reporting problem, etc., are true in *all* experimental disciplines. And the oft-claimed difference in statistical evidence provided by the so-called "hard" vs. "soft" sciences is a myth, as discussed in The Conscious Universe (HarperOne, 1997).

The one and only difference with parapsychology is that the underlying phenomena do not fit well with prevailing ideas about what consciousness is and what it is capable of. That one difference is enough to cause people to be more critical of the meta-analytical evidence, even though the actual techniques used are identical to those used in conventional domains.