Who is Killed by Police: Introducing Supervised Attention for Hierarchical LSTMs

Minh Nguyen, Thien Huu NguyenCOLING 2018

TL;DR

This paper introduces a novel deep learning approach for automatically detecting if a person mentioned in text was killed by police. By using hierarchical LSTMs and a new "supervised attention" mechanism, the model significantly improves accuracy over previous methods, achieving state-of-the-art results. This work is crucial for better understanding and tracking police killings through text analysis.

What Problem Does This Paper Address?

Identifying instances of police killings from text is a growing concern, especially with increased public attention on police shootings. While some prior work exists, it often relies on traditional machine learning with hand-crafted features or struggles to fully leverage the power of deep learning models. A key challenge is that a person's name might appear in many sentences, only some of which describe a police killing. Effectively sifting through this contextual information to pinpoint the relevant details is difficult. Existing methods, like those using Expectation Maximization (EM) with distant supervision, have limitations in handling the complexities of deep learning representations and require significant feature engineering.

Methods

The core of this paper's contribution lies in its novel deep learning architecture:

  • Hierarchical LSTMs: To handle the multiple sentences associated with a person's name (the "sentence container"), the authors employ a two-layer LSTM structure. The first layer processes words within each sentence, and the second layer processes the resulting sentence representations to form a container representation. This allows the model to learn from the sequential nature of words and sentences.
  • Supervised Attention Mechanism: This is the paper's main innovation. Traditional attention mechanisms in similar models can overemphasize the entity's name itself, neglecting crucial contextual words. To address this, they introduce "supervised attention." This mechanism uses "intuitive attention weights" derived from linguistic heuristics (semantical word lists and syntactical dependency paths) to guide the model's attention. Essentially, it biases the model to pay more attention to words deemed important by these external linguistic rules, helping it better capture the nuances of police killing events.
  • Semantical Aspect: Identifies important words by looking for "police," "killing," and their synonyms.
  • Syntactical Aspect: Leverages the shortest dependency paths between the entity name and "police" words in dependency trees to find crucial contextual words.

Datasets

The research utilizes the police killing dataset released by Keith et al. (2017). This dataset was created using distant supervision, where names from a corpus are matched against a database of police-killed victims (like the Fatal Encounters database) to label entities as positive (killed by police) or negative. The authors split the original training data into training and development sets for parameter tuning and then used the original training and test data for comparison with state-of-the-art models. No new benchmark dataset was introduced.

Key Insights and Ideas

  • Attention is Crucial, but Needs Guidance: The paper strongly demonstrates that attention mechanisms are vital for this task, but standard attention can be myopic, focusing too much on the target entity.
  • Linguistic Intuition Enhances Deep Learning: By injecting external linguistic knowledge (semantical and syntactical cues) into the attention mechanism, the model's ability to identify relevant contextual information is significantly boosted.
  • Syntactical Guidance Outperforms Semantical: The study found that using dependency paths (syntactical aspect) for guidance was more effective than relying solely on word lists (semantical aspect), especially for correctly classifying negative instances.
  • Eliminating Hand-Crafted Features: The proposed hierarchical LSTM model achieves competitive or superior performance without relying on the laborious hand-crafted features often required by traditional methods.

Key Results

  • The proposed H-LSTM+SynAtt model achieved state-of-the-art performance on the police killing detection task, outperforming previous best models like soft-LR and soft-CNN.
  • Specifically, H-LSTM+SynAtt achieved an absolute F1 score improvement of 3.3% over the best prior model (soft-LR) on the test data.
  • Both supervised attention models (H-LSTM+SemAtt and H-LSTM+SynAtt) showed improvements over the baseline H-LSTM (1% and 1.7% F1 score improvement, respectively), highlighting the benefit of supervised attention.
  • The models that excluded attention mechanisms (Mean+SynAtt, Mean+SemAtt) performed significantly worse, underscoring the importance of attention.

Limitations and Future Work

  • Limitations: The paper notes that the semantical attention (H-LSTM+SemAtt) can be biased by the presence of "police" and "killing" related words in negative sentences, potentially leading to false positives. The authors also mention that the dataset in Keith et al. (2017) might contain many sentences with these keywords, which could influence results.
  • Future Work: The authors plan to apply their proposed method to a real-world system and extend it to detect other types of events, such as protests and epidemics.

Summary updated

BibTeX
@inproceedings{nguyen2018who,
  author = {Minh Nguyen and Thien Huu Nguyen},
  title = {{Who is Killed by Police: Introducing Supervised Attention for Hierarchical LSTMs}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure