Extracting meaningful information from the chaotic stream of Twitter during breaking news events is a significant challenge in natural language processing. Traditional approaches rely on rigid "pipeline" architectures that process data in isolated stages. This paper, presented at IJCAI-17, introduces JEDS (Joint Event Detection and Summarization), a neural model that demonstrates how filtering noise, grouping tweets, and summarizing events can be learned together to improve overall performance.
The Problem: Pipeline Propagation Errors
Detecting and reporting events on Twitter typically involves three distinct sub-tasks:
- Filtering: Identifying relevant event-related tweets while discarding mundane chatter.
- Clustering: Grouping related tweets into specific event "buckets."
- Summarization: Selecting the most informative tweets to represent each cluster.
In a traditional pipeline, the output of the filter feeds the clusterer, which then feeds the summarizer. This structure suffers from two major flaws. First, errors in early stages—such as a false positive in filtering—propagate and degrade the performance of later stages. Second, these tasks are naturally interdependent; a tweet that serves as an excellent summary is, by definition, highly relevant to the event. Standard pipelines cannot exploit these bi-directional relationships.
The JEDS Architecture
The core innovation of JEDS is its ability to allow these three tasks to inform one another through shared representations and "neural stacking."
Shared Tweet Representation
Every tweet is first processed by a Long Short-Term Memory (LSTM) network to create a dense vector ($H$). This vector captures the semantic meaning of the tweet and serves as the common foundation for all subsequent tasks.
Neural Stacking and Joint Training
Instead of passing raw data between modules, JEDS passes features. The hidden layer of the Mention Detection (filtering) task is fed as an input to the Clustering task. Similarly, clustering features are fed into the Summarization network. Because the entire stack is differentiable, the model can back-propagate errors from the summarization stage all the way back to the filtering stage. This allows the model to learn that if a tweet is "summarizable," it is likely highly relevant.
For clustering, the model employs a Siamese Network to calculate similarity between tweets, determining whether a new tweet should join an existing cluster or initiate a new one.
Experimental Setup and Datasets
To evaluate the model, the authors collected and annotated two new datasets using the Twitter streaming API:
- Earthquake Domain: 47 events and ~12,000 tweets.
- DDoS Attack Domain: 170 events and ~17,000 tweets.
Data was manually annotated based on NIST Topic Detection and Tracking (TDT) rules to ensure high-quality ground truth for both event detection and summarization quality.
Key Results
JEDS consistently outperformed both traditional discrete pipelines and independent neural pipelines across multiple metrics. The joint training approach proved particularly effective at reducing the error rate in clustering while maintaining high summarization quality.
- Clustering Accuracy: Measured by $C_{min}$ (where lower is better), JEDS achieved 45.8 on the Earthquake dataset, significantly outperforming the popular AS12 stream-clustering model (60.9) and Locality Sensitive Hashing (66.7).
- Summarization Quality: Measured by ROUGE-1, JEDS scored 21.3, outperforming the state-of-the-art extractive summarizer LexRank (18.8) when used in a pipeline configuration.
Insights and Future Work
A critical finding of this research is the "Summarization Feedback" effect: training the model to recognize what an informative summary looks like actually improves its ability to filter out irrelevant noise in the initial detection phase. Furthermore, the study confirmed that neural representations (LSTMs) are superior to traditional "bag-of-words" features (TF-IDF) for short, slang-heavy social media text.
Despite these gains, the model remains purely extractive, meaning it selects existing tweets rather than generating original summary text (abstractive summarization). Future work may explore integrating abstractive components and moving toward "open-domain" detection that does not rely on pre-defined keywords.
Summary updated