Estimating conflict losses and reporting biases

Benjamin J. Radford, Yaoyao Dai, Niklas Stoehr, Aaron Schein, Mya Fernandez, Hanif SajidProceedings of the National Academy of Sciences 2023 · DOI

TL;DR

This paper tackles the challenge of accurately estimating military losses in conflicts, where official reports are often biased or incomplete. By developing a novel Bayesian statistical model that accounts for source-specific reporting biases and correlations between different loss types, the authors provide more reliable daily and cumulative loss estimates for the Russia-Ukraine conflict. This approach offers a crucial tool for understanding conflict dynamics and improving accountability.

What Problem Does This Paper Address?

Accurately quantifying casualties and fatalities in armed conflicts is vital for everything from conflict measurement and forecasting to holding parties accountable. However, obtaining reliable data is incredibly difficult. Belligerents have strong incentives to manipulate or conceal loss figures, while external observers often lack direct access to accurate information. This is starkly illustrated by the wildly varying estimates of losses in the ongoing Russia-Ukraine conflict. The "fog of war" and inherent biases in reporting create a significant gap between the reality on the ground and what is publicly known, making it challenging to assess the true cost of conflict.

Methods

The authors introduce a sophisticated, multivariate Bayesian model designed to estimate daily and cumulative losses. The core innovation lies in its ability to simultaneously:

  • Integrate diverse reporting sources: The model incorporates thousands of loss reports from various origins, including government sources, NGOs, and open-source intelligence (OSI).
  • Account for reporting biases: Crucially, the model treats source-specific biases as estimable parameters. This means it learns whether a particular source tends to over- or underreport losses, rather than assuming all reports are equally reliable.
  • Model structural correlations: The authors recognize that losses in different categories are often related (e.g., armored vehicle losses might correlate with personnel casualties). The model leverages these correlations to fill gaps in reporting for specific loss types.
  • Handle temporal scales: It integrates both daily and cumulative loss reports, allowing for a more complete picture over time.

The model uses Poisson and negative binomial distributions for loss counts and employs cubic basis splines to capture latent time series of expected daily losses, allowing for flexible temporal modeling.

Datasets

The study utilizes a newly constructed dataset comprising 4,609 reports of military and civilian losses related to the Russia-Ukraine conflict. These reports were aggregated from various sources and categorized into 21 types, including military and civilian fatalities, casualties, and equipment losses (e.g., tanks, helicopters, artillery). The dataset captures losses at both daily and cumulative levels, though cumulative reports constitute the vast majority. The authors categorize their reporting sources into seven groups: OSI, Russian sources, UK sources, Ukrainian sources, United Nations, US sources, and "other" sources.

Key Insights and Ideas

  • Bias is systemic, not random: The paper strongly suggests that reporting biases are not just occasional errors but systematic tendencies of different sources. Understanding and quantifying these biases is key to unlocking more accurate loss estimates.
  • The "fog of war" is quantifiable: By modeling the uncertainty around estimates, the paper demonstrates that even with imperfect data, we can derive meaningful insights into conflict dynamics.
  • Correlations are powerful: Leveraging the relationships between different types of losses (e.g., equipment vs. personnel) provides a robust way to infer missing data and improve overall estimates.
  • Public perception vs. reality: The paper implicitly highlights how official narratives and public reporting can diverge significantly from estimated realities, underscoring the need for independent, data-driven analysis.

Key Results

  • Higher Russian personnel losses: The model estimates that Russia suffered substantially more personnel losses than Ukraine. As of February 23, 2023, estimated Russian personnel losses were around 76,687, compared to Ukraine's 17,223.
  • Higher Russian fatality-to-casualty ratio: Russia also appears to have a higher fatality-to-casualty ratio (2.9:1) compared to Ukraine (4.9:1).
  • Equipment losses more balanced: Relative equipment losses between the two sides are estimated to be closer to parity.
  • Reporting bias patterns:
    • Both sides tend to overestimate the personnel losses of their opponent.
  • Russian sources underestimate their own personnel losses significantly (reporting only 0.3 losses for every actual loss).
  • Ukrainian sources overestimate Russian deaths by nearly double, though with less statistical certainty.
  • No systematic bias was found in Ukrainian reports of Ukrainian military deaths.

Limitations and Future Work

  • Data limitations: While the dataset is extensive, it is still based on publicly available reports, which inherently have limitations. The authors acknowledge that for some loss categories, the latest observed report may be many months prior to the estimation date (Feb 23, 2023).
  • Model assumptions: The model relies on specific statistical distributions (Poisson, negative binomial) and assumptions about the nature of biases. While principled, these are still simplifications of complex reality.
  • Future work: The authors suggest that their approach can be extended to other conflicts and types of losses. Further refinement of bias modeling and incorporating additional data sources could enhance accuracy. The ability to predict future losses based on these models is also a promising avenue.

Summary updated

Abstract

Determining the number of casualties and fatalities suffered in militarized conflicts is important for conflict measurement, forecasting, and accountability. However, given the nature of conflict, reliable statistics on casualties are rare. Countries or political actors involved in conflicts have incentives to hide or manipulate these numbers, while third parties might not have access to reliable information. For example, in the ongoing militarized conflict between Russia and Ukraine, estimates of the magnitude of losses vary wildly, sometimes across orders of magnitude. In this paper, we offer an approach for measuring casualties and fatalities given multiple reporting sources and, at the same time, accounting for the biases of those sources. We construct a dataset of 4,609 reports of military and civilian losses by both sides. We then develop a statistical model to better estimate losses for both sides given these reports. Our model accounts for different kinds of reporting biases, structural correlations between loss types, and integrates loss reports at different temporal scales. Our daily and cumulative estimates provide evidence that Russia has lost more personnel than has Ukraine and also likely suffers from a higher fatality to casualty ratio. We find that both sides likely overestimate the personnel losses suffered by their opponent and that Russian sources underestimate their own losses of personnel.
BibTeX
@inproceedings{radford2023estimating,
  author = {Benjamin J. Radford and Yaoyao Dai and Niklas Stoehr and Aaron Schein and Mya Fernandez and Hanif Sajid},
  title = {{Estimating conflict losses and reporting biases}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2023},
  doi = {10.1073/pnas.2307372120},
}
Figure