Limitations of the meta-analysis

The meta-analytical approach has its advantages. Unfortunately, it has also, like any statistical tool, its own weaknesses. The meta-analysis is plagued by several methodological problems (Glass et al., 1981; Stanley, 2001): the selection or publication bias, the presence of heterogeneity among studies and dependence of study result.

2.2.1 Selection and publication bias

The first drawback of the meta-regression approach is concerned with attaining a representative sample of the literature. Even if access to modern bibliographical tools (Econlit, for example) and the availability of many working papers through the web have greatly facilitated the research of pertinent primary studies, it remains very challenging to assess that a particular sample of studies appropriately represents the population of studies.

Meta-analysts are particularly afraid that the process of literature retrieval is such that the likelihood of sampling a study is correlated with the result it contains. This may be due to a restrictive sampling over time, within a country or a language or alternatively because of focus on a specific theoretical or modeling approach. In this regard, one principal aggravating factor is the possibility that the well-known published (and easily accessible) studies present a biased sample of what has been found by researchers. As Olkins (1990) points out, there might be a « statistical star wars » effect if journal editors tend to reject insignificant or disappointing results. Furthermore, the researchers may actually self-censor. Ashenfelter et al. (1999) claim that economists share theoretical presumptions and therefore tend to leave their contradictory results in the file-drawer. Consequently, available results will tend to overestimate the size and the significance of the effect size. Card and Krueger’s (1995) meta-analysis presents an example of the file-drawer problem. When examining the relationship between a change in minimum wage and the unemployment rate, they observed that a large number of studies found a negative employment effect with t-values close to the value of 2. The clear implication is that published studies are not a random draw of estimated minimum wage effects.

Many studies might also stay out of reach. This is the case of the consultancy studies undertaken by both the public and private sector since confidentiality problems might exclude them from available primary studies. Furthermore, when available, the results may not be presented objectively when they are used as vehicles of policy advocacy or simplified for the sake of public comprehension.

Most economic meta-analyses do not consider the potential emergence of a selection and publication bias⁸. However, in some cases, it is possible to test for its presence. Florax (2001) presents and illustrates the available procedures. Common techniques range from graphical funnel graph analysis⁹

8 This problem also happens in literature reviews.

9 This is a quasi-statistical technique, introduced by Light and Pillemer (1984), which relies on a graphical analysis where the effect site estimates are plotted on the horizontal axis and the sample size of the respective studies on the vertical axis.

Distortions of the funnel like shape (with the tip pointed up, and centered on the true effect size under the null hypothesis of no

(Card and Krueger, 1995) to estimating the number of additional studies that would be necessary in order to inverse the conclusion¹⁰ (fail-safe N, file drawer test) or the use of a two-stage Heckman selection model¹¹. Another approach aims to estimate two separate regressions for published and unpublished results or to include a dummy variable catching published studies. A final technique consists of using the model of publication bias developed by Hedges (1992). This model is based on the assumption that the probability of observing a study is a function of the p-value, whereby studies with lower p-values are more likely to be observed (Ashenfelter et al., 1999).

Overall, the importance of publication bias remains a controversial question, as the distinction between published and unpublished studies is time dependent (since numerous working papers may finally be published). Furthermore, the focus on publication bias should not be overemphasized since the whole selection process of the primary studies is finally at sake (Smith and Huang, 1995). In this regard, we have to note that the problem of representativeness is not specific to meta-analyses but to the whole non-experimental empirical literature.

2.2.2 Heterogeneity among studies

The second drawback addresses the comparability of the estimated effect size. This critique claims that meta-analyses are comparing and mixing “apples and oranges” and gives as an outcome an average value of their weight, perfume or color (Glass et al., 1981). It designates the fact that studies examining an identical question may not be directly comparable since their research design, type of data, estimator, functional forms and specification may differ.

This type of heterogeneity is not always straightforward. For instance, elasticity estimates may differ as for their estimation method (double logarithmic vs point estimates evaluated at the sample mean of prices and quantities) or time horizon (short-run vs long-run). These estimates are incomparable and these differences therefore have to be modeled in the meta-regression (by fixed or random effects).

Identically, the comparison of EKC estimates on logarithmic or linear specifications is problematic. The

”apples and oranges critique” constitutes a bigger concern in social and economic sciences since the degree of heterogeneity of the relevant studies is quite important¹².

The heterogeneity among primary studies might however also constitutes an advantage when one wishes to trace the influence of different specifications, data and research designs on the estimates. In other words, it might be a good thing to mix apples and oranges if one wants to generalize about fruit since studies that are exactly the same in all respects are actually limited in generalizability.

In addition to this substantive heterogeneity, the distribution of the effect tends to be heteroscedastic, because estimated effects sizes are based on studies with different sample sizes. We expect therefore

publication bias) may be taken as an indication that publication bias is present. The selection effect on the basis of significance and size is signaled by a graph that is skewed to the right or left, or with the lower center part missing (Florax, 2001).

10 See Rosenthal (1979).

11 See Heckman (1990) for a review.

12 In medicine and sciences, replication is frequent and empirical studies have to follow an established protocol. On the contrary, in economics, it seems to be a common desideratum of research that the investigator be original and innovative (Florax et al., 2002a). This consideration is very subjective. However, when one looks at the research motivations in the abstracts of published papers, the terms original, new contribution, new evidences are frequent. Furthermore, many studies put forward in their introduction or conclusion that “to the authors’ knowledge, the present study has not already been done”.

to find a positive relationship between the sample size and the statistical power or significance of the effect size. This heterogeneity may be treated adequately by specifying a fixed or random effect model (Jeppessen et al., 2002), the application of either a weighted regression (Cavlovic et al., 2000) or a heteroscedasticity robust variance estimator (Woodward and Wui, 2001).

A third aspect of heterogeneity among studies addresses the fact that the quality of the primary studies may differ. However, even if the meta-analytical approach can again weigh primary studies’

results according to their quality differences, the problem is that constructing objective quality indicators is inherently difficult. As Cooper (1986, 67) points out, even if justified by the analyst, the decision to weigh, include or exclude studies on a “a priori” basis requires the analyst to make an overall judgment of quality that is often too subjective to be credible. Economic meta-analyses have usually not considered this issue. In this regard, one might claim that literature reviews do this better.

However, even if subjective quality judgments are easier to formulate in a literature review, their adequacy remains controversial. Furthermore, literature reviews usually compare only a few studies and do not have to order the whole sample of primary studies according to their quality (as a meta-analytical quality judgment would require)¹³.

The theoretical literature on the meta-analysis offers several options in order to select studies according to their quality. The most usual one is the published-unpublished distinction, which recommends selecting only the published primary studies since the reviewing process guarantees a high qualitative level. However, this procedure might generate a publication bias. Furthermore, publication may be a poor indicator of quality since there are now far more academic journals, which while offering the scope for easier diffusion of research findings, may also make it easier for poor quality research to appear in print (Van den Bergh et al., 1997). Another possibility is to let subjective reviewing considerations distinguish between high quality and low quality studies and to control if the latter give systematically alternative results. Woodward et Wui (2001) proceed this way by classifying the studies in three groups of different quality according to the apparent coverage of the database used as well as the methodological and econometric consistency. The meta-regressions then examine if the outcomes of high quality studies are systematically different¹⁴.

2.2.3 Independence of results

A final problem of the meta-analytical approach concerns the assumption of independence of the observations. In experimental sciences, this assumption may be defended since the tradition of replication allows the meta-analyst to select one estimate per study without running into degrees of freedom problems. In economics, the number of available studies is generally limited and each study examines competitive specifications. The meta-analyst must therefore select several observations from the same study. As these observations are usually derived from the same database, the assumption of independence of the observations does not hold (Florax et al., 2002a).

13 The literature reviews of Ekins (1997, 194-195) on the EKC illustrate this point. The authors favor the results on air pollution obtained by Selden and Song (1994) and disregard the study of Panayotou (1993). Quality differences explain this choice.

However, they are not compared to other studies on air pollution.

14 Note that other meta-analysts have selected studies according to other considerations. For example, Nelson (1980) considers only the « best » studies, but does not give more precise criteria for this selection. Button and Weyman-Jones(1994) reject the studies that do not pass some arbitrary statistical cut-off point.

If observations are dependent, one will expect that results emanating from one piece of original research will be more similar than those coming from different studies. In econometric terms this will manifest itself as correlation in the error terms associated with the estimates from the same studies.

Most meta-analyses disregard this problem even if it might be shown that this form of heteroscedasticity will have a downward bias on the standard errors estimated on the coefficients, erroneously increasing the coefficients’ apparent significance. Varying solutions to overcome this statistical problem have been proposed¹⁵. The solution of Loomis and White (1996) is straightforward as he considers only one observation per study and rejects the others¹⁶. Espey (1998) examines the correlation matrix between error terms for studies offering more than 4 observations. A dummy variable is then introduced for every subset of observations whose errors are strongly correlated¹⁷. Doucouliagos (1995) takes into consideration the average of the observations of each primary study.

Finally, Day (1999) accounts for the clustering of estimates by study allowing observations to be dependent inside a cluster (i.e., a primary study) and independent between clusters.