What is meta-analysis? What are the benefits of using it in research?

Question

What is the downside to using meta-analysis?

Does using meta-analysis detract from the validity of statistical analyses?

I have heard that it is often used in Psychological and Parapsychological studies as well as when doing analyses of census data. If this is true, why would it be used in these types of studies more often than in the physical sciences?

Answer 1

First, this is exactly the kind of scientific question I like seeing in here.  :)

A meta-analysis is a study of studies, literally. It is a statistical analysis where the results of many studies are gathered and analyzed together in order to derive a conclusion from the sum total. Why is this approach taken?  See the excellent text from Tufts university below. Basically, meta-analysis becomes necessary when (1) no large scale high-quality studies are available and/or (2) the varying quality of the studies that are available requires extra analysis in order to extract a statistically useful conclusion from them.

For example, meta-analysis is often used in the field of medicine for interpreting clinical data which may deliver statistically inconclusive results for one reason or another. However, it's well known that this approach is neither rigorous nor reliable. In fact, in a recent paper discrepancies were noted between meta-analysis and subsequent large, randomized, controlled clinical trials in as many as 35 percent of the cases studied (see the New England Journal of Medicine reference).

One of the biggest problems with meta-analysis is the file-drawer effect, where published studies are strongly biased in favor of reporting positive results while negative results are rarely published, skewing the meta-analysis in favor of a positive outcome. The other big problem is knowing how to cope with the varying quality of the studies and how to manipulate the analysis so that the results of each study are "weighted" according to their merit. This is not a rigorous procedure and it varies from researcher to researcher. Moreover, it would theoretically be possible to do one hundred experiments with small samples and all with negative outcomes, while a meta-analysis of the same data would produce results that are, mathematically speaking, statistically significant. Many poor quality studies put together do not make one large good quality study. The researcher performing the meta-analysis has to be keenly aware of this and impose the correct statistical controls to avoid this pitfall.  Since this answer is getting long already, I'll leave it to the reader to peruse the references below.

Aside from interpreting medicinal clinical studies, you really won't find meta-analysis used in the physical sciences since the world of physics is predictable, repeatable and amenable to testing. Even the world of quantum physics is predictable and testable, statistically speaking. Due to the inherent unreliability of meta-analysis it is mostly used for generating hypothesis instead of supporting theories. It is notable here that parapsychology is unique in the fact that it is the only area of research which actually relies on meta-analysis almost exclusively for support.

Answer 2

Meta analyses attempt to sort of collate data from many different experiments.  The problem with this is that although the data aren't worthless, the fact that they were gathered inconsistently tends to cast a dubious shadow on any results.  It's impossible to determined the integrity of all experiements or to create a standard among them.  As such, the results should be taken with a grain of salt.

Answer 3

a lot of the problem with most studies of paranormal and other implausible things is the small numbers.

Small groups being investigated increase the chances of false positive results.

A meta-analysis is a statistical way to try and combine several small studies of the same thing into one larger study in an attempt to deal with that problem

The Cochrane database is a very large one for medical studies.  But they take all comers...from JAMA to Journal of Crystal Healing.

They arena't a perfect solution, as there is also the chance of magnifying the errors and raising the false positive rate

Answer 4

Meta-analysis is a quantitative way of estimating the size of experimental effects and the degree to which those effects are independently repeatable. It is used extensively in the social, behavioral, medical and biological sciences, because effects in those realms tend to be small and erratic, and as such, no one study is likely to conclusively demonstrate anything. When properly conducted, it provides a rigorous way of reviewing all known evidence for well defined subsets of experimental data. Many critiques of meta-analysis suggest that through statistics one can prove anything, but this is a misunderstanding both of meta-analysis and of statistical inference. That meta-analytical articles appear in many major journals is evidence that the methods are sound when applied appropriately.

Meta-analysis is used in parapsychology for the same reasons it is used elsewhere: It addresses the question of effect size and repeatability in complex systems, where high variability is unavoidable.

The physical sciences tend to work with systems that are far more stable than living systems, so repeatability is typically easier. However, even within the physical sciences something like meta-analysis is commonly used to set the value of physical constants, because all experiments, however well conducted, always contain some error.

Meta-analyses conducted for classes of parapsychological experiments show that some effects are real (meaning, not due to chance), they are repeatable, and they are not adequately explainable as selective reporting, or variations in experimental quality.

Answer 5

So is that the properly name for a controlled experiment.It you conduct a controlled experiment under very strict conditions then you arrive at a absolute answer.The answer must be correct as all other possibilities are ruled out.I used this method doing telekinesis.It actually worked.