DEFT 2018: Attention sélective pour classification de microblogs (DEFT 2018 : Selective Attention for Microblogging Classification )

Charles-Emmanuel Dias, Clara Gainon de Forsan de Gabriac, Patrick Gallinari, Vincent GuigueCORIA-TALN-RJC (DeFT) 2018 · OpenReview

Abstract: Charles-Emmanuel Dias, Clara de Forsan de Gainon Gabriac, Patrick Gallinari, Vincent Guigue. Actes de la Conférence TALN. Volume 2 - Démonstrations, articles des Rencontres Jeunes Chercheurs, ateliers DeFT.

2018.

Introduction

This paper presents a hierarchical recurrent neural network with a custom attention mechanism designed specifically for classifying noisy microblogs (tweets). By explicitly isolating and weighting hashtags and user mentions via a learned gating mechanism, the model effectively handles the chaotic nature of Twitter data without requiring heavy preprocessing, achieving highly competitive results in the DEFT 2018 challenge.

Problem Statement

Applying Natural Language Processing (NLP) to microblogs like Twitter is notoriously difficult. Tweets are short and plagued by approximate spelling, excessive or missing punctuation, slang, and emojis. Furthermore, they contain platform-specific structural elements—namely, hashtags (#) and direct mentions (@)—which carry a high density of semantic information. Historically, achieving good performance on tweets required massive, complex preprocessing pipelines and heuristics to clean the text before feeding it to traditional classifiers (like SVMs). The authors aim to bypass this manual feature engineering by building a deep learning architecture that can natively ingest raw, noisy text and specifically leverage the high-value information hidden inside hashtags and mentions.

Methodology

The authors build upon the Hierarchical Attention Network (HAN) architecture, introducing two critical modifications tailored to the unique structure of microblogs:

1. Character-to-Word Hierarchy

Standard HANs encode words into sentences and sentences into documents. However, tweets often lack reliable sentence structure and contain significant spelling variations. To address this, the authors shift the hierarchy down: their model uses a Bidirectional GRU with attention (referred to as an RBA) to encode characters into words, and a second RBA to encode those words into a complete tweet representation. This character-level foundation provides robustness against typos and slang.

Hierarchical architecture from characters to tweet representation.

2. Custom Hashtag/Mention Attention Gate

The core novelty of the paper is a selective masking mechanism at the word-to-tweet level. The network explicitly identifies words starting with # or @, summing their hidden states to create a specific "metadata" representation ($e_m$). A learned linear interpolation gate ($\beta$, utilizing a sigmoid function) then combines this metadata with the standard attention-weighted representation of the tweet ($e_s$).

Selective attention mechanism for hashtags and mentions.

This dual-pathway approach allows the standard attention mechanism to focus on traditional linguistic markers while ensuring the vital context provided by hashtags and mentions is preserved.

Experimental Setup and Datasets

The model was trained and evaluated on the DEFT 2018 Challenge Dataset, which consists of French-language tweets focused on public transportation in the Île-de-France region. The evaluation involved two primary tasks:

  • Task 1 (Topic Classification): Binary classification to determine if a tweet is about transportation.
  • Task 2 (Sentiment Analysis): Multi-class classification assigning polarity (Neutral, Negative, Positive, MixPosNeg) to transport-related tweets.

Results and Discussion

The model proved highly competitive in the DEFT 2018 challenge against 11 other teams:

  • Task 1: The model achieved an accuracy of 83.048%, earning 2nd place overall.
  • Task 2: The model achieved an accuracy of 65.824%, placing 6th. The authors noted a slight training error in Task 2; they estimate that without it, the model's true accuracy would have been approximately 68%, which would have secured 3rd place.

Key insights from the study demonstrate that by treating the space character as the only reliable delimiter and building word embeddings dynamically from characters, the network learns to abstract away from severe grammatical and spelling errors without external tools. Minimal preprocessing, combined with the right inductive biases (hierarchy and specific attention), can achieve state-of-the-art results on raw social media text.

Conclusion and Future Work

While the character-level encoding provides significant robustness, the authors suggest that future work could involve dedicated, separate representations for hashtags and mentions—untethered from their constituent letters—to potentially yield even greater performance gains. Additionally, further investigation into the routing and weighting of emojis could enhance the model's understanding of sentiment in microblogs.

Summary updated

BibTeX
@inproceedings{dias2018deft,
  author = {Charles-Emmanuel Dias and Clara Gainon de Forsan de Gabriac and Patrick Gallinari and Vincent Guigue},
  title = {{DEFT 2018: Attention sélective pour classification de microblogs (DEFT 2018 : Selective Attention for Microblogging Classification )}},
  booktitle = {CORIA-TALN-RJC (DeFT)},
  year = {2018},
  url = {https://openreview.net/forum?id=owHd0tTPukF},
}
Figure