How Should We Study District Judge Decision-Making?

Pauline T. Kim, Margo Schlanger, Christina L. Boyd, Andrew D. Martin

Abstract

Understanding judicial decision-making requires attention to the specific institutional settings in which judges operate. The choices available to judges are determined not only by the law and facts of the case but also by procedural context. The incentives and constraints shaping judges’ decision-making will vary depending on, for example, whether they have a life-appointment or are elected; whether they hear cases alone or with colleagues; and whether and under what circumstances their decisions might be altered, overturned, or undone by the actions of others. The basic insight that the institutional context matters has led to increasingly sophisticated studies of how strategic interactions among Supreme Court justices, among branches of government, and within a judicial hierarchy shape judicial decision-making. But studies of federal district judges—the nearly one thousand judges who compose 78% of the federal judiciary and superintend 79% of its cases—have not matched this sophistication. Instead, much of the existing empirical work on federal district courts has failed to take account of the institutional setting in which those judges operate.

The "Sophistication Gap" in Judicial Research

Federal district judges are the workhorses of the American legal system. Comprising nearly 80% of the federal judiciary, they handle the vast majority of cases that enter the system. Yet, empirical legal scholarship often treats these trial judges as "mini-Supreme Court justices." This creates a significant "sophistication gap": while researchers have developed complex strategic models to understand the Supreme Court and appellate levels, they frequently apply those same models to district courts without accounting for the vastly different institutional realities of the trial bench.

Appellate judges generally act once on a fixed record to resolve a specific legal question. In contrast, district judges are "managerial." They oversee cases from filing to finish, making hundreds of small, procedural decisions that never culminate in a formal "opinion." By focusing only on published opinions, scholars are looking at a tiny, non-random slice of the legal system, leading to potentially skewed conclusions about how the law actually functions.

From "Opinionology" to "Docketology"

To bridge this gap, this paper advocates for a shift from "opinionology"—the study of written judicial opinions—to "docketology." This new approach focuses on the entire lifecycle of a case as recorded in court dockets.

Comprehensive Event Tracking

Rather than treating a case as a single data point (e.g., "who won?"), docketology tracks the entire "event history" via PACER (Public Access to Court Electronic Records). This allows researchers to see every procedural move, from discovery disputes to scheduling orders.

Relational Database Architecture

Standard "flat" databases (simple rows and columns) are insufficient for the messy reality of trial courts. One case might involve two motions, while another involves two thousand. The authors propose a multi-table database structure that links a main "case" table to an "events" table, capturing the sequence, timing, and context of every litigation milestone. This prevents researchers from "comparing tadpoles to frogs" by ensuring that decisions are analyzed within their specific stage of litigation.

The Reality of Managerial Judging

Under Federal Rule of Civil Procedure 16, district judges are active managers. Their primary power often lies not in "declaring the law" through a final judgment, but in setting schedules and discovery limits. These procedural interventions create a settlement feedback loop. A judge’s ruling on a minor motion to amend a complaint or a discovery request can drastically shift the "settlement value" of a case, making litigation too expensive or risky for one party and forcing a resolution outside of formal adjudication.

The Lethal Threat of Selection Bias

The paper highlights a critical flaw in traditional research: the "3% Rule." Previous data suggests that only about 3% of judicial actions are accompanied by written opinions. Relying solely on these opinions introduces massive selection bias for several reasons:

  1. Ideological Skew: Published district court opinions are systematically more "liberal" than unpublished ones. Studies based only on Westlaw or Lexis likely overestimate the impact of a judge's personal ideology on the bench.
  2. The Rarity of Adjudication: In the federal system, adjudication is the exception, not the rule. For example, in 2006, roughly 47% of federal cases were abandoned or settled without any form of formal adjudication.
  3. Outlier Cases: Cases that survive long enough to produce a published opinion are outliers. In the other 97% of judicial actions, procedural rules and factual records are likely the dominant factors, rather than the political or religious background of the judge.

Case Study: EEOC Litigation

The authors applied these "docketology" principles to a study of nearly 4,000 employment discrimination cases filed by the Equal Employment Opportunity Commission (EEOC) between 1996 and 2006. By supplementing EEOC internal records with manual and automated scraping of PACER dockets, the researchers were able to build a more granular picture of how these cases move through the system, identifying which judicial interventions most frequently triggered settlements.

Conclusion and Future Directions

While "docketology" is labor-intensive and requires parsing messy, free-form text entered by court clerks, it is essential for a realistic understanding of the judiciary. Future research should move toward hazard models (survival analysis) to predict the distribution of settlements over time based on the timing and nature of judicial interventions. To truly understand how district judges decide cases, we must stop looking only at their final words and start looking at their everyday actions.

Summary updated

Abstract

Understanding judicial decision-making requires attention to the specific institutional settings in which judges operate. The choices available to judges are determined not only by the law and facts of the case but also by procedural context. The incentives and constraints shaping judges’ decision-making will vary depending on, for example, whether they have a life-appointment or are elected; whether they hear cases alone or with colleagues; and whether and under what circumstances their decisions might be altered, overturned, or undone by the actions of others. The basic insight that the institutional context matters has led to increasingly sophisticated studies of how strategic interactions among Supreme Court justices, among branches of government, and within a judicial hierarchy shape judicial decision-making. But studies of federal district judges—the nearly one thousand judges who compose 78% of the federal judiciary and superintend 79% of its cases—have not matched this sophistication. Instead, much of the existing empirical work on federal district courts has failed to take account of the institutional setting in which those judges operate.
BibTeX
@inproceedings{kim2009how,
  author = {Pauline T. Kim and Margo Schlanger and Christina L. Boyd and Andrew D. Martin},
  title = {{How Should We Study District Judge Decision-Making?}},
  booktitle = {Open},
  year = {2009},
}
Figure