A meta-analysis combines results from multiple program evaluations to assemble a composite of the evidence about what works to obtain a desired outcome. Meta-analyses can be particularly powerful when there are repeated evaluations of a single intervention (or minor variations of an intervention) in different settings or with different samples of the target population. Even when the studies evaluate slightly different interventions, a meta-analysis can provide powerful evidence.
Typically, the meta-analysis author sets specific inclusion criteria and then conducts a wide-ranging literature search to find all of the studies, published and unpublished, that fit those criteria. Like the evaluations they combine, most meta-analyses focus on causal evidence to show what works to achieve a certain outcome.
A meta-analysis is usually conducted on a group of similar programs targeting comparable outcomes. However, even similar programs might vary in terms of what the exact intervention is, how it is implemented, and how outcomes are measured. The meta-analysis inevitably conceals some of this variation, which can make it challenging to determine exactly what it is about the programs' features that achieves or fails to achieve a desired outcome.
The methods of a meta-analysis are fairly demanding and typically exclude studies with weaker designs. CrimeSolutions places a premium on strong designs; consequently, meta-analyses that set a low evidence standard often score lower or are simply excluded from the clearinghouse.
At the time of this writing, CrimeSolutions includes 50 "practices," which present results from one or more meta-analyses on a given topic. The meta-analyses focus on causal evidence for specific criminal or juvenile justice outcomes, such as desistance or stopping truancy. Studies included in a single meta-analysis tend to focus on the same outcomes, and their programmatic features, such as dosage and target population, tend to be similar. Frequently, however, the programs differ in important ways. For instance, some programs may use CBT as the main intervention, while others in the same meta-analysis may use CBT in conjunction with other intervention strategies. Some programs may provide a higher dosage of the intervention or target a lower-risk population. (Some meta-analyses might include additional analyses to determine these differences and their potential impact on outcomes.)
Thus, although meta-analyses assemble a great deal of evidence in a single calculation of effectiveness, important details often lie beneath the CrimeSolutions practice rating. It is always a good idea to read the entire practice profile on CrimeSolutions to get a complete understanding of what works and what does not.
About This Article
This article appeared in NIJ Journal Issue 277, September 2016.
Sidebar to the article Does Cognitive Behavioral Therapy Work in Criminal Justice? A New Analysis From CrimeSolutions by Thomas Feucht and Tammy Holt.