Abstract
Detecting real-world incidents via social media is significantly hindered by regional variance, where models trained on one city often fail in another due to local linguistic differences. This paper evaluates the generalization capabilities of various feature sets—ranging from simple n-grams to complex sentiment and stylistic markers—across ten diverse cities. The results demonstrate that basic word and character n-grams provide highly effective and robust baselines, often outperforming more sophisticated features. Minor performance gains were observed only with the addition of Named Entity Recognition, suggesting that for cross-regional incident detection, model simplicity is key to effective generalization.
Introduction
Social media platforms like Twitter have become vital tools for emergency responders, often providing reports of incidents faster than official channels. However, a major technical hurdle remains: regional variance. A model trained to detect traffic accidents in Seattle may struggle in London due to differences in slang, street naming conventions (e.g., "I-90" vs. "M25"), and local writing styles. Since labeling thousands of tweets for every new city is prohibitively expensive, there is a critical need for generalized models—classifiers that can be trained on data from one region and still perform accurately in a completely new location. This research investigates which features help a model ignore local "noise" and focus on universal signals of an incident.
Experimental Methodology
The researchers conducted a comprehensive feature engineering "bake-off," implementing a wide array of features categorized into baselines (word and character n-grams), sentiment analysis, Named Entity Recognition (NER), stylistic/Twitter-specific patterns, and string similarity measures. They compared two primary machine learning algorithms: Support Vector Machines (LibLinear) and Naive Bayes (NB).
To test generalization, the study employed a "leave-one-city-out" evaluation protocol. A model was trained on data from one city and tested on the remaining nine, repeating this process for all ten cities in the dataset to ensure the results were not biased toward a specific geographic region.
Data and Benchmark
The study utilized a benchmark consisting of over 20,000 manually labeled tweets across 10 diverse cities, including Boston, Chicago, New York, London, Sydney, and Dublin. Tweets were classified into four categories: Crash, Fire, Shooting, and Not Incident Related. The data was collected over three distinct periods between 2012 and 2014 to ensure temporal variety. Notably, the researchers found that only 14% to 23% of tokens were shared between different city datasets, highlighting the inherent difficulty of the generalization task.
Feature Performance Analysis
The "bake-off" revealed several key insights regarding feature effectiveness:
- N-gram Dominance: Simple word and character sequences were remarkably difficult to beat. They captured the core vocabulary of incidents effectively enough that more complex features often added nothing but noise.
- Complexity vs. Generalization: Many intuitive features, such as sentiment analysis or complex string similarity, actually hindered generalization. These features tended to overfit to the specific ways people in one city express emotion or structure their sentences.
- Character N-grams for Noise: For the Naive Bayes classifier, character-level features (e.g., "cras", "rash") proved superior to word-level features. This robustness is likely due to their ability to handle the typos, abbreviations, and "Twitter-speak" common in social media.
Results
The best generalized models achieved an overall F1-measure of > 83% across all ten cities. While LibLinear generally provided better average performance (F1 = 83.32) compared to Naive Bayes (F1 = 80.48), the margin between simple baselines and complex feature sets remained slim.
As shown in the results, using just the top 5,000 word unigrams yielded an F1 of 82.87. The addition of Named Entity Recognition (NER) and @-mention features—one of the few sophisticated sets that actually helped—only increased the F1 score to 83.48, a marginal improvement of approximately 0.6%.
Conclusion and Future Work
The findings suggest that for incident classification, more features are not always better. Simple n-gram baselines provide a strong foundation for cross-regional generalization. The authors acknowledge limitations such as keyword bias in data collection and the focus on English-speaking cities.
Future work will explore Semantic Abstraction, which involves replacing specific entities (e.g., "I-90") with generic tags (e.g., [HIGHWAY]). This approach aims to allow models to learn universal patterns, such as "Crash on [HIGHWAY]," regardless of the specific local geography, further improving the ability to detect incidents in unseen regions.
Summary updated