Abstract: Gun violence is a critical public health and safety concern in the United States. There is considerable variability in policy proposals meant to curb gun violence, ranging from increasing gun availability to deter potential assailants (e.g., concealed carry laws or arming school teachers) to restricting access to firearms (e.g., universal background checks or banning assault weapons). Many studies use state-level variation in the enactment of these policies in order to quantify their effect on gun violence. In this paper, we discuss the policy trial emulation framework for evaluating the impact of these policies, and show how to apply this framework to estimating impacts via difference-in-differences and synthetic controls when there is staggered adoption of policies across jurisdictions, estimating the impacts of right-to-carry laws on violent crime as a case study.
Introduction
Gun violence remains one of the most pressing public health crises in the United States. However, evaluating the effectiveness of policies designed to mitigate this violence is a complex statistical challenge. Because researchers cannot randomly assign laws to states, they must rely on observational "panel data"—repeated measurements of states over time. This paper, authored by Eli Ben-Michael and colleagues, provides a rigorous methodological roadmap for researchers navigating this landscape. By framing gun policy evaluation as a "policy trial emulation," the authors demonstrate how modern causal inference techniques can address the limitations of traditional observational studies.
The Challenge of Observational Policy Evaluation
Estimating the effect of a law is notoriously difficult due to two primary factors:
- Staggered Adoption: States pass laws at different times. Simple "before vs. after" comparisons are often misleading because they fail to account for broader national trends occurring simultaneously.
- Violation of Assumptions: Traditional methods, such as Two-Way Fixed Effects (TWFE) regression, rely on the "parallel trends" assumption—the idea that treated and control states would have followed the same trajectory in the absence of the law. In reality, a state passing a gun law might already be experiencing a unique surge or decline in crime, which biases the results.
A Framework for Policy Trial Emulation
The authors advocate for a design-focused approach that treats observational data as an emulation of a randomized trial. They highlight three key methodological pillars to improve the credibility of these evaluations:
Stacked Difference-in-Differences (DiD)
Instead of a single global regression, this method "stacks" individual datasets for each policy change. It compares states that adopted a law to a specific set of control states that had not yet adopted it. This approach avoids "forbidden comparisons," where early adopters are inadvertently used as controls for late adopters, a common flaw in older studies.
Synthetic Control Method (SCM)
When a single state lacks a natural "twin," SCM creates a "synthetic" version of that state by taking a weighted average of other states. For example, a "Synthetic Ohio" might be constructed from a weighted combination of Rhode Island, Hawaii, and California to better mirror Ohio’s pre-policy crime trends.
Augmented SCM
Standard SCM can fail if the "donor" states do not perfectly match the target state's baseline crime levels. The authors utilize a bias-correction approach that incorporates fixed effects to adjust for these differences, allowing for more credible counterfactuals even when the match is imperfect.
Case Study: Right-to-Carry (RTC) Laws
To demonstrate these methods, the authors analyze the impact of Right-to-Carry (RTC) laws—specifically "shall-issue" or "permitless" carry regimes—using data from the FBI Uniform Crime Reports (UCR) and CDC mortality data from 1977–2014.
Methodological Insights
- Lumping vs. Splitting: Researchers must choose between granular policy categories (splitting) and broader categories (lumping). While splitting provides detail, the authors argue that lumping policies into a general "Right-to-Carry" category increases statistical power, making it easier to detect meaningful effects.
- The Importance of Pre-trends: Using Ohio as an example, the authors show that while a standard DiD model might suggest a crime increase following a law, an "event study" plot reveals that Ohio's crime was already trending upward relative to other states before the law was enacted. Identifying these pre-existing trends is crucial for debunking untrustworthy results.
- Cohort-Level Aggregation: Rather than analyzing one state at a time, the authors suggest finding synthetic controls for "time cohorts" (groups of states that passed laws in the same year). This reduces noise and improves the model's fit.
Key Results and Uncertainty
Applying the Augmented SCM to RTC laws, the authors found that the impact on violent crime is negligible in the immediate five years following adoption. However, a gradual upward trend emerges in the long term. Ten years after adopting RTC laws, the model estimates a roughly 15% increase in violent crime.
Despite this estimate, the authors emphasize the presence of significant uncertainty. The 95% confidence intervals remain wide and include zero, suggesting that while the point estimate is positive, the limitations of the underlying UCR data make it difficult to draw definitive conclusions.
Limitations and Future Directions
The paper concludes by identifying several hurdles for the field:
- Data Quality: FBI UCR data is subject to under-reporting and inconsistent practices across decades, which can pollute statistical models.
- Rare Outcomes: Current SCM techniques are optimized for continuous rates and struggle with "rare events," such as homicides in smaller jurisdictions.
- Policy Interactions: Laws do not exist in a vacuum. Disentangling the effects of "compound treatments"—such as a state passing both "Stand Your Ground" and RTC laws simultaneously—remains a major challenge.
- Temporal Resolution: Moving from annual data to monthly or quarterly data could provide more pre-treatment data points, leading to more robust synthetic controls.
Summary updated