Improving NIJ's Peer Review Process: The Scientific Review Panel Pilot Project

by Thomas E. Feucht and Phyllis Newton

NIJ is piloting a new grant application peer review process with standing panels designed to improve the consistency, fairness and transparency of peer review at the Institute.

Pilot Solicitations

The following solicitations will be part of the pilot project:
  • Basic Scientific Research to Support Forensic Science for Criminal Justice Purposes
  • Applied Research and Development in Forensic Science for Criminal Justice Purposes
  • Evaluating the Impact of the NIJ Body Armor Program
  • Replication Research on Sexual Violence Case Attrition
  • Longitudinal Data on Teen Dating Violence: Postdoctoral Fellowship
  • The Impact of Different Safety Equipment Modalities on Reducing Correctional Officer Injuries
  • Research on Policing
  • Determining the Relationship Between Stress and Unexplained In-Custody Deaths
  • Research and Evaluation on Violence Against Women: Sexual Violence, Stalking, and Teen Dating Violence
  • Research and Evaluation in Justice Systems
  • Research on the Impact of Technology on Policing Strategies in the 21st Century

Two key events in the "life course" of the National Institute of Justice occurred in 2010. In June, the National Research Council (NRC) released its evaluation of NIJ, Strengthening the National Institute of Justice; and in July, John Laub became the Director of NIJ — the first NIJ director to have a Ph.D. in criminal justice.

The release of the evaluation and Laub's appointment put NIJ in a good position to reinvigorate its science mission, one of NRC's core recommendations.[1]

NRC noted the need for NIJ to do a better job in its research program development and planning and to put in place more rigorous processes and policies related to research. Perhaps no operational issue drew more attention than the peer review of applications for research grant awards. Citing serious concerns about NIJ's current peer review processes, NRC's evaluation called for significant improvements in how peer review is conducted in order to safeguard the science and to ensure a fair, transparent and competitive process for making research grants.

Peer review lies at the heart of grantmaking. It guarantees an independent assessment of the scientific merit of the proposed research. Rigorous, fair and transparent reviews by external scientific reviewers help ensure that the competitive grant process works as it should, that the best science gets funded, and that standards of scientific excellence are consistently applied to all grant applications.

Peer Review in the Past at NIJ

For more than two decades, NIJ's peer review process involved assembling small committees (usually three or four reviewers) for each review cycle — a typical way to conduct anonymous peer reviews. But because the panels were selected anew each year, problems could arise with consistency from one year to the next. Applicants who were offered an opportunity to revise and resubmit, for example, had their applications reviewed the second time by a completely different panel. In addition, assembling a committee could be challenging when deadlines were short.

The NRC's evaluation characterized NIJ's peer review as "very weak," and urged the Institute to look to other science agencies, like the National Science Foundation and the National Institutes of Health, for good peer review models.

NIJ took the recommendation to heart and has begun a pilot program to strengthen its review processes.

Peer Review in the Future at NIJ

Starting in the review cycle for fiscal year 2012, NIJ will establish a total of five Scientific Review Panels in the following topic categories:

  • Criminal justice systems
  • Violence and victimization
  • Forensics (two panels)
  • Science and technology

Each panel will consist of 12 scientists and six practitioners. Scientific members will serve for overlapping three-year terms to provide continuity, consistency and experience. Practitioner members will serve one-year terms. The panelists, recognized authorities in their field, will be nominated by other researchers and practitioners. Final selection will be made by the appropriate NIJ Office director. The names of the panelists will be posted following the announcement of grant awards on NIJ.gov.

NIJ anticipates the need to include ad hoc panelists who can review applications that require specialized expertise. Some solicitations, for example, are likely to generate proposals for research in narrowly defined areas of study, such as some engineering and biological sciences. In such cases, NIJ will invite ad hoc members with applicable expertise to serve on the Scientific Review Panels. In most cases, ad hoc members will serve for a single review cycle.

Steps in the Review Process

NIJ's grant application review process can be summarized into four major steps. The members of NIJ's pilot Scientific Review Panels will be involved in two of the four steps:

Step 1. NIJ staff will verify that applications meet the basic minimum requirements (BMR) as stated in the solicitation. BMR verification is conducted to ensure that the documents necessary for effective scientific review are included in the application. As a rule, an application that includes a program narrative, a budget narrative, a budget detail worksheet and curriculum vitae for key personnel will meet BMR.

Step 2. Each application will be read by three members of a Scientific Review Panel (the lead and two co-leads) who will assess the application's technical merit and write narratives of their reviews. They will give each application a response of "forward to the full panel" or "do not forward to the full panel" based on scientific merit, including such factors as statement of problem, project design and implementation, capabilities/competencies, budget, impact/outcomes and evaluation, and dissemination strategies.

Step 3. All 18 members of a Scientific Review Panel will have the opportunity to review every application and participate in the consensus review. Panels will meet for two or three days once a year, during the same week every year. For each forwarded application, the lead and co-lead reviewers will give a brief overview, present their scores, outline the strengths and weaknesses, and lead the discussion. Following the discussion, each panelist will score the applications using a written ballot.

Step 4. NIJ staff will receive and review the scores and narrative written by the lead and co-leads. The staff will then make recommendations to the Director, who will make final decisions based on the scores as judged by the reviewers, how the proposed projects fit into NIJ's priorities and research agendas, and the availability of funding.

Scoring the Applications

At the meeting of the full panel, each panelist will have an opportunity to discuss the application and the lead and co-leads' narrative before giving the application an overall score for quality.

Scores will be based on a 5-point scale from 5 (excellent) to 0 (poor), in which intermediate values are treated as equal intervals on the scale.

Score Range Adjectival Equivalent
4.5 to 5.0 Excellent
3.5 to 4.0 Good
2.5 to 3.0 Acceptable
1.5 to 2.0 Minimally Acceptable
0 to 1.0 Poor

The Scientific Review Panel's overall score for each application will be a consensus. The panel therefore must resolve disparities that arise among reviewers' overall scores. If a difference of opinion remains, a minority report will be written and submitted to NIJ.

To ensure transparency of the review process and to aid applicants in developing effective applications, applicants will receive their final scores and written technical reviews.

As Director Laub stated in his response to the NRC report, "At the core of a strong science agency is a rigorous and fair peer review process. All grants, for instance, must be awarded as the result of a fair, open, and competitive peer review process." NIJ's new Scientific Review Panels are an important step toward ensuring that NIJ's precious research resources are invested in only the best research proposals and that each proposal submitted to NIJ receives a fair and scientifically sound review.

NIJ Journal No. 268, October 2011
NCJ 235892

About the Authors

Thomas E. Feucht is NIJ's Executive Senior Science Advisor. Phyllis Newton is the Director of NIJ's Office of Research and Evaluation.

Back to the top.

For More Information

Notes

[1] Visit the Director's Corner on NIJ.gov. Read John Laub's vision for NIJ in which he emphasizes the need for NIJ to respond fully to the findings and recommendations of the National Research Council's evaluation of NIJ

Date Modified: February 29, 2012