Systematic Reviews and Meta-Analysis as Research Summaries

Theoretical Research

The evidence-based movement sought to make research findings more accessible to clinicians. Andrew D. Banasiewicz draws on established research-synthesis methods, including work by the Center for Evidence-Based Management, in the theoretical-research component of his 3E framework. Figure 7.7 outlines the approach.

FIGURE 7.7 Extracting Evidence from Theoretical Research.

Banasiewicz uses theoretical research to cover conceptual and empirical work. In Figure 7.7, the 3E framework includes the empirical studies because their findings can be checked against observational or experimental data. Conceptual work develops or refines ideas. Those ideas may generate testable hypotheses, but they do not supply the same kind of evidence as the tests themselves.

The framework divides empirical research into observational studies, experiments, and research summaries. In the organizational settings discussed here, observational studies often analyze transaction or communication data, supplemented by details about people or firms. Such data can reveal associations, but observational research is not limited to records generated independently of a study. Experimental studies deliberately vary conditions to test hypotheses. Research summaries assess the findings of multiple studies.

Research is distributed across academic, commercial, governmental, and other organizations. Researchers may independently investigate similar questions using different samples, measures, and analytical methods. Their findings can differ because of those choices, chance, or genuine differences between settings. Research summaries help assess the combined evidence and identify where results agree or remain uncertain.

A systematic review (SR) searches for studies addressing a defined question, appraises them, and synthesizes their findings using explicit methods. Compared with an informal narrative review, this makes selection and interpretation easier to inspect. A meta-analysis (MA) uses statistical methods to combine comparable results and examine variation between studies. It requires suitable numerical information, which may limit the studies that can be pooled. A meta-analysis can form part of a systematic review, but numerical pooling is not inherently more rigorous: its value depends on the question, included studies, and analytical choices.

Systematic Reviews

Leveraging the Cochrane Collaboration developed conceptual framing, SR can be more formally defined as an assessment of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research; it may include the use of statistical methods in the form of MA to analyze and summarize the results of the included studies. The stated goal of SR is to minimize reviewer bias by adopting a replicable and a transparent process, which also leaves an audit trail of reviewers’ choices and conclusions. A key part of the ‘systematic’ aspect of those reviews is a general process, graphically summarized in Figure 7.8.

The seven-step SR process begins with carefully delimiting the research question of interest, followed by the selection of, first, databases to be used in literature search, followed by the selection of individual articles. The database selection step can be seen as a reflection of the size of the scholarly publishing universe and the resultant difficulty associated with clearly targeting search efforts, considering that some industry estimates count nearly 29,000 individual scholarly, peer-reviewed journals. And while most of those thousands of journals can be searched using curated databases such as Thomson-Reuters’ Web of Knowledge, Elsevier’s Scopus, ABI/INFORM, or SCImago, and typically only a small subset of journals might be of interest, the search may nonetheless yield large volumes of qualifying studies.

Once the focal databases have been selected, the more tedious and difficult process of finding individual articles ensues. Research articles are typically indexed based upon author-supplied keywords, and more open-ended database search operations tend to make use of simple bag-of-words text mining functionalities that disregard grammar, and even word ordering. Thus, results of literature searches tend to yield large volumes of ‘raw’ matches, out of which only a handful ultimately fit the search criteria. The search and selection aspect of the overall SR process can be further systematized by approaching it as a series of distinct decisions of identification-screening-eligibility-inclusion. Search-identified records are screened for duplicates and topical fit, then assessed based on reviews of abstracts, following which a more in-depth eligibility assessment is undertaken by examining the full texts of articles, ultimately arriving at the final set of records.

FIGURE 7.8 Systematic Review.

From the perspective of evidence gathering, the next two steps – extraction of data from studies and assessment of the quality of individual studies – constitute the heart of the SR process. The process is structured and it addresses review-related considerations, prior to delving into analyses of the content of selected studies. The former commonly includes an outline of the rationale for the review and the stated review objective, expressly addresses methods used in the review process and points to protocols guiding the methods used, clearly spells out eligibility for inclusion, search, and selection criteria, as well as data collection processes, inclusive of methods used for assessing risk of bias of individual studies. Aiming to minimize reviewer bias, reviews of content of individual studies emphasize clear summaries of the main findings, the efficacy of the supporting evidence (e.g., statistical significance, goodness-of-fit, etc.) while also noting any limitations, considered at study and outcome level. Lastly, reviewers provide a general interpretation of results of each study while also pointing out sources of funding supporting the review process, along with the role played by fund providers. It should be noted that the analysis and assessment stage of SR can encompass MA, or the use of statistical techniques to integrate the results of individual studies, however, that is an optional, rather than a persistent part of the review process.

The final stages combine the assessments and interpret the body of evidence. A useful synthesis explains the overall findings, their uncertainty, and where differences between studies limit their application.

Meta Analyses

MA can be defined as a set of statistical procedures designed to assimilate results of related empirical studies with the goal of deriving generalizable effect estimates. As a solution to the problem of pooling data across multiple related studies to reconcile differences across reported results, meta-analytic-like approaches were proposed in the early 1900s, but as a distinct and defined methodology, MA emerged in the mid-1970s.13 The key premise of the newly proposed methodology was that past research in an area of interest could be consolidated by standardizing measures from individual studies and computing effect size across studies. Further work on meta-analytic methods offered means for combining probability values across studies to enable assessment of the overall level of statistical significance. Since then, numerous applications of MA showed that the systematic use of established statistical procedures to amalgamate results of individual research studies produces more valid and reliable estimates of observed relationships than would be possible by relying on singular studies. As used today, the general MA approach encompasses quantitative assessment of effect size and the level of statistical significance, in addition to also including methodological guidance on weighting of individual studies; as noted earlier, it can be conducted as a part of the broader SR initiative, or as a standalone project.

As is the case with most methodologies, to yield dependable outcomes, meta-analytic initiatives need to adhere to established procedures. Those include the following: (1) selection of studies to be included in the analysis, which need to be both representative and topically related, (2) validity of individual studies, all of which need to demonstrate adequate degree of empirical rigor (e.g., sample size, analysis, inferences), and (3) methodological homogeneity across studies, or the use of similar or comparable research methods and effect estimation approaches across studies.

Considering that MA can be nested within SR, the two process flows share some distinct similarities, as graphically depicted in Figures 7.8 and 7.9. Both begin with the same initial steps of defining research question, selecting of databases, and selecting of individual articles, and although the next step – extraction of data from studies – is also notionally the same for MA and SR, methodologically and procedurally it entails very different means and outcomes.

MA requires well-defined numerical inputs, including effect sizes: measures such as correlations, mean differences, or odds ratios. Interpretation depends on the measure and its scale; an unstandardized regression coefficient cannot be compared directly across arbitrarily different units. The magnitude and direction of an effect 14 must be interpreted for the particular measure. Before pooling, reviewers need to establish that the estimates address a sufficiently comparable question. They should examine variation between studies and choose a model appropriate to their assumptions. A single pooled estimate can hide important differences; a random-effects analysis may estimate an average effect and between-study variation. A homogeneity test alone does not decide whether pooling is meaningful.

Meta-analysis does not remove bias in the underlying studies or guarantee that their results apply elsewhere. Incomplete searches, selective publication, weak study designs, and incompatible outcomes can all distort a pooled result. Statistical synthesis is useful when it makes these limits clearer rather than concealing them behind a precise average.

FIGURE 7.9 Meta-Analysis.