General Methodology
The Researchers

The research for Justice Index 2016 was carried out in 2015 under the supervision of NCAJ’s staff by 37 pro bono attorneys from five law firms. These law firms are:

Kirkland & Ellis LLP
Morgan, Lewis & Bockius LLP
O’Melveny & Myers LLP
Patterson Belknap Webb & Tyler LLP
Simpson Thacher & Bartlett LLP
The Initial Outreach to States

To determine laws, rules, and practices across the United States, NCAJ invited the Chief Justice and Chief Court Administrator of each state, the District of Columbia, and Puerto Rico to complete a Justice Index Questionnaire. NCAJ then worked with teams of pro bono attorneys to conduct a quality assurance review of the responses for accuracy. The origin of the access to justice indicators contained in the Questionnaire is described here.

To determine the count of civil legal aid attorneys for the Attorney Access Index, NCAJ relied on data provided by the Legal Services Corporation (“LSC”) and on outreach to non-LSC-funded legal aid organizations.

In 45 of the 52 jurisdictions, officials provided NCAJ with their responses to the Justice Index Questionnaire. The work required to assemble these responses was substantial and NCAJ and its law firm volunteers are grateful to the court systems, judges, attorneys, and staff who participated in preparing and providing each response.

For each of the 45 responsive jurisdictions, the pro bono attorneys, working in teams from each firm, reviewed the responses and conducted a quality assurance review that included a variety of steps such as corresponding with court officials, reviewing citations and sources, and conferring with other volunteer attorneys and NCAJ staff.

For the seven jurisdictions that did not provide responses to the Justice Index Questionnaire – Alaska, Iowa, Kansas, Michigan, New Hampshire, New Jersey, and South Dakota – NCAJ and the pro bono attorneys carried out original research on the respective states’ laws, policies, and practices.

The Quality Assurance Review Process

The pro bono attorneys reviewed the responses and sources provided by each jurisdiction and set forth their recommendations as to whether to award or deny credit with respect to each indicator.

NCAJ staff conducted at least two rounds of review of the proposed determinations and citations, during which NCAJ worked with the pro bono attorneys to invite state officials to provide clarification or further information where necessary. NCAJ then notified officials in each jurisdiction via email of the proposed findings and, in instances where NCAJ considered further clarity necessary, explained the bases for the proposed findings, and invited comments, questions, and supplemental materials.

NCAJ conducted an additional round of review and correspondence with state officials who responded to the proposed findings. As a final step, NCAJ emailed the states a non-public version of Justice Index 2016 before making the website publicly available.

Our Process and Standard for Making “Yes” and “No” Findings

For each indicator, state court officials were invited to respond “Yes” if the state had adopted the law or practice on a statewide level, and to provide a source to support the “Yes” response.

As stated on the Questionnaire provided to the states, the default source required for a “Yes” finding was “a statewide statute, rule, regulation, appropriation, or other written guidance.” We made exceptions to the documentation requirement for certain indicators where written sources were less likely to be available, such as training questions or questions about employees or offices designated to perform specific tasks. For these indicators, we generally considered a source sufficient if it (i) identified the court official who oversees the practice or oversaw the training; (ii) includes the official’s contact information; and (iii) includes some level of specific background information, such as when the training was held or what authority designated the employee or office to perform the specific task.

NCAJ conducted several rounds of quality assurance review in which we provided our state correspondents with opportunities to change or supplement sources to establish that the state had adopted the practice in question. We requested written, public sources where available, explaining that in most cases they would be required a for a “Yes” finding. If NCAJ determined that a source established the presence of the statewide practice or law, a “Yes” finding was made. All final “Yes” findings receive the full number of points allocated to that indicator based on the Justice Index weighting system.

If a state did not respond “Yes” for an indicator, or if a state responded “Yes” but did not provide sources that were found to be sufficient during our quality review process, a “No” finding was made. It is important to note that a “No” finding means only that a “Yes” finding was not established; it does not mean that NCAJ has affirmative support for a finding that the state has not adopted the subject law or practice. Nor does it mean that the state is not, in actuality, currently carrying out the practice in some form or in some place – as stated above, most “Yes” findings required written citations; we generally considered unwritten policies insufficient. Our reason for this is because, if a practice can be stopped at any time without the notice and public attention that comes from having to first amend a written document, then it is not a best practice. In recent years, budget cuts have resulted in at times drastic reductions in court services, and it is the services based on informal, unwritten policies that are likely the first to go. Changes in justice system administration, or in administration outside the justice system, may also put certain unwritten policies in jeopardy. For these reasons, we believe that requiring written sources for practices promotes accountability, predictability, and fair notice.

Addition of the “No*” Finding to Justice Index 2016

In addition to the “Yes” and “No” findings in the Best Practices tables, this year we added “No*” findings that carry an asterisk (“*”). The asterisk denotes that we have included sources, citations, or other information that we believe will be of potential interest to Justice Index visitors because these materials illuminate what the state is doing that is relevant to the specific indicator, even though the practice itself, or the supporting documentation, was not sufficient to establish a “Yes” finding. Several “No*” findings were made for made for the type of unwritten policies referenced above, i.e., for indicators where a state official explained the practice, but did not point to a written source authorizing, requiring, or encouraging the practice. To maximize the information provided in the Justice Index and to maintain transparency, we have attempted to include all substantive citations and comments provided by the states despite our negative finding on that indicator.

A Note on Responsibility
NCAJ stresses that it, alone, is responsible for Justice Index 2016. No state, district, territory, institution, organization, or other entity or person other than NCAJ is responsible for the Justice Index, including for its evaluative indicators, research process, or process for integrating findings and producing scores. NCAJ is also responsible for all specific findings, except and unless in instances in which NCAJ has expressly acknowledged that it relied on specific entities or sources for final findings, as follows:
  • National Coalition for a Civil Right to Counsel (“NCCRC”) – NCAJ included in the Justice Index four “civil right to counsel” indicators and their respective findings as drawn from the national interactive map maintained by NCCRC at civilrighttocounsel.org/map. The indicators appear in the Justice Index categories of Self-Representation (#32 and #33) and Disability Access (#12 and #13). NCAJ has included these findings in the Justice Index with the explicit permission of NCCRC.
  • American Bar Association (“ABA”) – NCAJ incorporated into the Self-Represented category (#28) the ABA’s findings as to whether a state has an Access to Justice Commission or “other access to justice entity.” These findings and the links to all of the Access to Justice Commissions appear at http://www.americanbar.org/groups/legal_aid_indigent_defendants/initiatives/resource_center_for_access_to_justice/atj-commissions/commission-directory.html.
Overall Guidance

NCAJ Executive Director David Udell, Senior Attorney and Justice Index Director James Gamble, and Staff Attorney Aaron Sussman are responsible for Justice Index 2016, including the vision, methodology, findings, quality assurance review process, presentation, and other dimensions. Aaron Sussman led the research (and several other) initiatives for Justice Index 2016, providing support and direction to the pro bono law firms and their attorneys, as described above.