The National Institute of Justice Randomized Controlled Trial Challenge in Criminal Justice Agencies

On this page find:

Section I. Overview

The National Institute of Justice (NIJ) was created in 1968 to facilitate research and scientific inquiry to help state, local and tribal governments address crime problems and challenges in the American criminal justice system. Of the various scientific methods of inquiry available, randomized controlled trials (RCTs) have the greatest likelihood of producing sound evidence because of the ability of random assignment to isolate a specific treatment of interest from all of the other factors that influence any given outcome.

RCTs are the scientific gold standard for evaluating programs. They are pervasive in medicine and marketing as part of the standard process for making decisions and advancing the field. The criminal justice system has been slow to adopt RCTs as the standard process for testing programs and could do more with RCTs to determine whether a strategy or intervention makes a meaningful difference.  For example, the Coalition for Evidence-Based Policy noted that when the Institute of Education Sciences assessed 120 education programs using RCTs, they found that 90 percent of those education programs had little or no effect. They also found that 60 percent of proposed medical interventions that had already been through some initial screening were not effective when evaluated with an RCT.

To encourage the use of RCTs in the criminal justice field, NIJ is issuing this Challenge for timely and innovative RCTs that address meaningful criminal justice problems. NIJ wants to encourage criminal justice agencies to use rigorous research methods to craft solutions to the problems they face. This competition hopes to create incentives for criminal justice agencies to use low-cost RCTs as a standard and straightforward approach to answering their questions and conducting their day-to-day business operations. For example, agencies could use RCTs to help answer questions such as:

  • Should agencies have eight, 10 or 12 officers per supervisor?
  • Should employees be required to go through training recertification every six months or every 12 months?
  • What happens if the workload is increased by 10 percent per employee versus decreased by 10 percent?
  • Do swift and certain sanctions reduce probation violations?
  • Does participation in a correctional education program reduce inmate misconduct?
  • Do mentoring programs impact re-arrest rates in the six months following release?

For the purposes of this Challenge, the exact question is not as important as developing the culture of rigorous evaluation in the decision-making process. NIJ will consider original and distinct RCTs, which could be straightforward and relatively simple or highly complex. Importantly, contestants may not propose RCTs that are currently ongoing in an agency or department.

Prizes will be awarded to the first five agencies to complete five successful RCTs. Contestants may submit more than five RCTs.

Please note: All contestants are required to comply with Department of Justice (DOJ) regulations on confidentiality and human subjects protection in the conduct of their RCTs. See “Other Requirements for OJP Applications”. See also the NIJ Web page for Human Subjects and Privacy Protection. RCT proposals should include a description of how human subjects and privacy protection will be addressed in the project plan, including a very brief discussion of human subjects exemptions as appropriate for individual RCTs.  Proposals lacking such information will not be considered. Where research involving human subjects is involved, Institutional Review Board review and approval/exemption is required, as well as a privacy certificate.

Section II. How to Enter

Application Instructions

See How to Apply for instructions on using the OJP Grants Management System.

Submission Period

The RCT Challenge submission period begins on July 30, 2014, and ends when all five prizes have been awarded or on September 30, 2016. Entries submitted before or after the designated periods will not be reviewed. Contestants must submit their entries through the Office of Justice Programs’ Grants Management System (GMS) via "The National Institute of Justice Randomized Controlled Trial Challenge in Criminal Justice Agencies" announcement. Registration and entry are free.

Submission and Approval Process

The submission and approval of all proposed RCTs will be conducted as follows:

Step 1 – Register a Planned RCT

The contestant submits each proposed RCT through GMS. Each proposed RCT must be submitted separately in GMS. Contestants may propose ideas for RCTs that vary from less complex to more complex as long as they are relevant to the agency’s mission and address a meaningful criminal justice problem.

All entries shall include a proposal to conduct an RCT.

Each proposed RCT shall: 

  • Be described in not more than five pages double-spaced in 12-point font with 1-inch margins.
  • Include a description of how human subjects and privacy protection will be addressed in the proposed project. Each RCT proposal must be submitted in PDF format.
  • Include a cover page (submitted in PDF format and not counted against the appropriate page limit) that shall include the title of the RCT, and the contestant’s name and complete contact information (i.e., name, address, telephone number, and e-mail address).
  • Include an abstract not to exceed 400 words (not counted against the appropriate page limit) that shall include the following:
    • Goals and Objectives: State the anticipated relevance to criminal justice practice and policy.
    • Subjects: If applicable, describe the number of subjects in the study, their characteristics (e.g., age, gender, race/ethnicity) and other pertinent attributes.
    • Research Design and Methods: Describe the methods to be used for conducting the proposed RCT(s) including a clear study period, the type of data, collection strategies, instruments, study sites, and other methods or procedures.
    • Analysis: Summarize the techniques proposed for data analysis.

No changes may be made to a proposed RCT once it has been approved by NIJ (as described in Step 3 of this section). However, contestants may submit a new RCT entry. Any entry exceeding the appropriate page limits will be disqualified.

Step 2 – Panel Review of the Admissibility of the RCTs

A panel will review the admissibility of each RCT entry to determine whether the following elements are included:  

  • The entry describes an RCT that is relevant to the mission of the subject agency.
  • The entry addresses a meaningful criminal justice problem.
  • The entry includes a description of the randomization of treatment and control groups.
  • The entry demonstrates that the design and sample size give the RCT sufficient power (i.e., power is at least 80 percent for detecting a medium effect size of 0.5 at α=0.05). For example, at least 64 subjects would be needed per group for a two-group t-test, or 34 subjects would be needed per group for a paired t-test analysis.

Proposed RCTs will be submitted to the panel for approval in the order received.

Step 3 – NIJ Approval of Proposed RCTs

Upon NIJ's approval of a proposed RCT, NIJ will notify the contestant that the project activities outlined in the proposed RCT may proceed. If an RCT entry fails to meet the criteria specified by NIJ and is not approved, contestants will receive feedback from NIJ with an option to revise and re-submit the entry. NIJ will notify each contestant of its decision to approve or reject each proposed RCT.

Because the time required for review of each proposed RCT may differ based on complexity or other factors, RCTs may not be approved in the order received.

Step 4 – Submission and Award Process

Upon completion of each RCT, contestants will be required to submit the RCT through GMS so NIJ can determine if it was conducted successfully. The submission should consist of a final summary overview of research results and the RCT data set(s). The final summary overview should not exceed five pages (double-spaced in 12-point font with 1-inch margins) and should address the purpose of the RCT, subjects involved (if applicable), RCT design and methods, data analysis, and findings. As part of its submission, contestants also will be required to submit the RCT dataset(s). An RCT submission will be judged successful by NIJ if:

  • The implementation of the RCT is consistent with the original entry.
  • The submission includes the dataset(s) with sufficient information to allow replication of the results.

If the RCT is judged to be successful, one point will be awarded to the contestant.

Completed RCTs will be submitted to the panel for judging in the order received.

Because the time required for review of each completed RCT may differ based on complexity or other factors, judging may not be completed in the order submitted.

Step 5 – Announcement of the RCT Challenge Winner

The submission, approval, and point process for RCT entries will be repeated and the first contestant to earn five points for successful completion of five qualified RCTs in the same criminal justice agency will be declared the winner.

Because the time required for review of each completed RCT may differ based on complexity or other factors, judging may not be completed in the order submitted.

Section III. Important Dates

  • Proposed RCT submission and approval period begins: July 30, 2014.
  • The Challenge period ends: September 30, 2016.

These dates are tentative. If five organizations complete five RCTs each in advance of the closing dates, NIJ may change the proposed and completed RCT submission dates.

Section IV. Judging Process

The RCT Challenge will be judged by a distinguished panel of judges with expertise in policing, courts, corrections, probation, parole, and research methods. All approved RCTs will be judged according to the criteria listed in Step 4 and a leaderboard will be maintained at NIJ’s website to track the progress of all contestants. The recommendations of the judges are advisory and the  Director of NIJ or his designee will make the final award determinations.

Section V. Prize

The winner of the Challenge will be the first contestant to complete five RCTs successfully. The RCT Challenge prize of $100,000 will be awarded to the first place winner upon successful completion of the fifth RCT. A prize of $50,000 each will be awarded to the second, third, fourth and fifth place RCT Challenge contestants. In the case of a team entry, the team will be considered a contestant and receive any prize money as a group.  All prizes will be awarded at the discretion of the NIJ Director or his designee. The award of any prize is subject to the availability of appropriated funds.

Section VI. Other Rules and Conditions

(A) Eligibility

Contestants in the Challenge may include : (1) a single criminal justice agency seeking to conduct at least five separate RCTs; or (2) an organization or individual resident of the 50 United States, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, the Northern Mariana Islands, Guam, and American Samoa working with a participating agency to conduct at least five separate RCTs.[1] A researcher may not partner with multiple agencies to achieve the five RCTs.  Contestants may submit or participate in the submission of more than one entry. Employees of NIJ and individuals or entities listed on the Federal Excluded Parties list are not eligible to participate. Employees of the federal government should consult with the Ethics Officer at their place of employment prior to submitting an entry for this RCT Challenge. The RCT Challenge is subject to all applicable federal laws and regulations. Submission of an entry constitutes a contestant's full and unconditional agreement to all applicable rules and conditions. Eligibility for the prize award(s) is contingent upon fulfilling all requirements set forth herein.

(B) General Warranties and Conditions

  1. Release of Liability. By entering the RCT Challenge, each contestant agrees to (1) comply with and be bound by all applicable rules and conditions, and the decisions of NIJ, which are binding and final in all matters relating to this RCT Challenge; and (2) release and hold harmless NIJ and any other organizations responsible for sponsoring, fulfilling, administering, advertising or promoting the RCT Challenge, and all of their respective past and present officers, directors, employees, agents and representatives (collectively, the "Released Parties") from and against any and all claims, expenses and liability arising from or relating to the contestant’s entry or participation in the RCT Challenge, and the contestant’s acceptance, use or misuse of the prize or recognition.

    The Released Parties are not responsible for (1) any incorrect or inaccurate information, whether caused by contestants, printing errors or by any of the equipment or programming associated with or used in the RCT Challenge; (2) technical failures of any kind, including, but not limited to malfunctions, interruptions or disconnections in phone lines or network hardware or software; (3) unauthorized human intervention in any part of the entry process or the RCT Challenge; (4) technical or human error that may occur in the administration of the RCT Challenge or the processing of entries; or (5) any injury or damage to persons or property that may be caused, directly or indirectly, in whole or in part, from the contestant's participation in the RCT Challenge or the contestant's receipt, use or misuse of any prize. If for any reason a contestant's entry is confirmed to have been deleted, lost or otherwise destroyed or corrupted erroneously, the contestant's sole remedy is to submit another entry in the RCT Challenge.
  2. Termination and Disqualification. NIJ reserves the authority to cancel, suspend or modify the RCT Challenge, or any part of it, if any fraud, technical failures or any other factor beyond NIJ’s reasonable control impairs the integrity or proper functioning of the RCT Challenge, as determined by NIJ in its sole discretion. NIJ reserves the authority to disqualify any contestant it believes to be tampering with the entry process or the operation of the RCT Challenge or to be acting in violation of any applicable rule or condition. Any attempt by any person to undermine the legitimate operation of the RCT Challenge may be a violation of criminal and civil law; should such an attempt be made, NIJ reserves the authority to seek damages from any such person to the fullest extent permitted by law. NIJ’s failure to enforce any term of any applicable rule or condition shall not constitute a waiver of that term.
  3. Intellectual Property. By entering the RCT Challenge, contestants warrant that (1) they are the authors or authorized owners of the entry; (2) the entry is wholly original with the contestants (or is an improved version of an existing solution that the contestants are authorized legally to enter in the RCT Challenge); (3) the submitted entry does not infringe any copyright, patent or any other rights of any third party; and (4) the contestants have the legal authority to assign and transfer to NIJ all of the rights, title and interest (past, present, and future) under copyright and other intellectual property law, for all material included in the RCT Challenge proposal that may be held by the contestants or the legal holder of those rights. Each contestant agrees to hold the Released Parties harmless for any infringement of copyright, trademark, patent or other real or intellectual property right, that may be caused, directly or indirectly, in whole or in part, from the contestant's participation in the RCT Challenge.
  4. Publicity. By entering the RCT Challenge, each contestant consents, as applicable, to NIJ’s use of his/her/its name, likeness, photograph, voice, and/or opinions, and disclosure of his/her/its hometown and state for promotional purposes in any media worldwide, without further payment or consideration.
  5. Privacy. Personal and contact information submitted through www.challenge.gov is not collected for commercial or marketing purposes. Information submitted throughout the RCT Challenge will be used only to communicate with contestants regarding entries or the RCT Challenge.
  6. Compliance with Law. By entering the RCT Challenge, each contestant guarantees that the entry complies with all federal and state laws and regulations.
  7. Specific Disqualification Rule. If the announced winner of the RCT Challenge prize is found to be ineligible or is disqualified for any of the reasons listed under “Other Rules and Conditions: Eligibility,” NIJ may make the award to the next runner up instead, as previously determined by the NIJ Director.
  8. Prize Disbursement and Requirements. Prize winners must comport with all applicable laws and regulations regarding prize receipt and disbursement. For example, NIJ is not responsible for withholding any applicable taxes from the prize award.
  9. Rights Retained and Assigned by Contestants and RCT Challenge Winners
    • Except as provided herein, all legal rights in any materials or products submitted in entering the RCT Challenge are retained by the contestant or the legal holder of those rights. Entry in the RCT Challenge constitutes express authorization for DOJ staff and NIJ’s selected panel of judges to review and analyze any and all aspects of submitted entries, including any source code and any trade secret or proprietary information contained in or evident from review of the RCT Challenge entry.  
    • Upon acceptance of any Challenge prizes, the winning contestants grant a worldwide, royalty-free, non-exclusive, perpetual and irrevocable right to NIJ to use, and authorize others to use, the winning submissions and all requisite data sets (in whole or in part, including in connection with derivative works) for federal purposes.

Note

[1] NIJ strongly encourages potential RCT contestants who lack in-house research capabilities to establish a partnership with an independent researcher or a research agency/institution.

Date Created: July 8, 2014