Abstract: Charles-Emmanuel Dias, Clara de Forsan de Gainon Gabriac, Patrick Gallinari, Vincent Guigue. Actes de la Conférence TALN. Volume 2 - Démonstrations, articles des Rencontres Jeunes Chercheurs, ateliers DeFT.
2018.
Introduction
This paper presents a hierarchical recurrent neural network with a custom attention mechanism designed specifically for classifying noisy microblogs (tweets). By explicitly isolating and weighting hashtags and user mentions via a learned gating mechanism, the model effectively handles the chaotic nature of Twitter data without requiring heavy preprocessing, achieving highly competitive results in the DEFT 2018 challenge.
Problem Statement
Applying Natural Language Processing (NLP) to microblogs like Twitter is notoriously difficult. Tweets are short and plagued by approximate spelling, excessive or missing punctuation, slang, and emojis. Furthermore, they contain platform-specific structural elements—namely, hashtags (#) and direct mentions (@)—which carry a high density of semantic information. Historically, achieving good performance on tweets required massive, complex preprocessing pipelines and heuristics to clean the text before feeding it to traditional classifiers (like SVMs). The authors aim to bypass this manual feature engineering by building a deep learning architecture that can natively ingest raw, noisy text and specifically leverage the high-value information hidden inside hashtags and mentions.
Methodology
The authors build upon the Hierarchical Attention Network (HAN) architecture, introducing two critical modifications tailored to the unique structure of microblogs:
1. Character-to-Word Hierarchy
Standard HANs encode words into sentences and sentences into documents. However, tweets often lack reliable sentence structure and contain significant spelling variations. To address this, the authors shift the hierarchy down: their model uses a Bidirectional GRU with attention (referred to as an RBA) to encode characters into words, and a second RBA to encode those words into a complete tweet representation. This character-level foundation provides robustness against typos and slang.
2. Custom Hashtag/Mention Attention Gate
The core novelty of the paper is a selective masking mechanism at the word-to-tweet level. The network explicitly identifies words starting with # or @, summing their hidden states to create a specific "metadata" representation ($e_m$). A learned linear interpolation gate ($\beta$, utilizing a sigmoid function) then combines this metadata with the standard attention-weighted representation of the tweet ($e_s$).
This dual-pathway approach allows the standard attention mechanism to focus on traditional linguistic markers while ensuring the vital context provided by hashtags and mentions is preserved.
Experimental Setup and Datasets
The model was trained and evaluated on the DEFT 2018 Challenge Dataset, which consists of French-language tweets focused on public transportation in the Île-de-France region. The evaluation involved two primary tasks:
- Task 1 (Topic Classification): Binary classification to determine if a tweet is about transportation.
- Task 2 (Sentiment Analysis): Multi-class classification assigning polarity (Neutral, Negative, Positive, MixPosNeg) to transport-related tweets.
Results and Discussion
The model proved highly competitive in the DEFT 2018 challenge against 11 other teams:
- Task 1: The model achieved an accuracy of 83.048%, earning 2nd place overall.
- Task 2: The model achieved an accuracy of 65.824%, placing 6th. The authors noted a slight training error in Task 2; they estimate that without it, the model's true accuracy would have been approximately 68%, which would have secured 3rd place.
Key insights from the study demonstrate that by treating the space character as the only reliable delimiter and building word embeddings dynamically from characters, the network learns to abstract away from severe grammatical and spelling errors without external tools. Minimal preprocessing, combined with the right inductive biases (hierarchy and specific attention), can achieve state-of-the-art results on raw social media text.
Conclusion and Future Work
While the character-level encoding provides significant robustness, the authors suggest that future work could involve dedicated, separate representations for hashtags and mentions—untethered from their constituent letters—to potentially yield even greater performance gains. Additionally, further investigation into the routing and weighting of emojis could enhance the model's understanding of sentiment in microblogs.
Summary updated