TL;DR
Researchers from IIIT Hyderabad have developed a massive new Twitter corpus for Hindi, Bengali, and Telugu—three widely spoken but "resource-scarce" languages. By bypassing standard API limitations and using emojis as natural sentiment labels, they’ve created a dataset of over 700,000 tweets to fuel better sentiment analysis and emoji prediction models for the Indian subcontinent.
What Problem Does This Paper Address?
While Twitter is a goldmine for Natural Language Processing (NLP), the research landscape is heavily skewed. Over 50% of tweets are in English, and consequently, most sentiment analysis tools are built for English.
Languages like Hindi, Bengali, and Telugu have hundreds of millions of speakers, yet they are considered "resource-scarce" in the AI world because they lack large, high-quality labeled datasets. Existing methods for building these datasets often face hurdles: Twitter’s official API limits historical searches to just seven days, and geolocation-based filtering misses users who don't share their GPS data or use desktop computers. This paper aims to bridge that gap by providing a scalable methodology to build large-scale corpora for these underserved languages.
Methods
The authors developed a pipeline that moves beyond the restrictive official Twitter Search API. Their approach involves three main steps:
- Keyword Extraction: They identified the 1000 most frequent words for each target language (using sources like
1000mostcommonwords.com). These words act as anchors to ensure the scraper finds "natural" discourse. - Query-Based Scraping: Instead of the standard API, they used a Python-based "twitterscraper" that crawls web search results. This allowed them to bypass temporal limits and gather tweets from any point in history. To build a dataset specifically for emoji prediction, they paired frequent words with specific emojis in their search queries (e.g.,
<❤️, की>). - Deduplication and Cleaning: To ensure data quality, they used hashmaps to track tweet IDs and the first 30 characters of tweet text, filtering out bot-generated spam and repetitive celebrity birthday wishes. They also created "Cleaned" versions of the data by stripping out URLs, mentions, and foreign-language symbols.
Datasets
The primary contribution of this paper is the Twitter Corpus of Resource-Scarce Languages.
- Total Tweets: 713,155
- Language Breakdown:
- Hindi: 396,478 tweets
- Telugu: 178,841 tweets
- Bengali: 137,836 tweets
- Structure: The data is split into tweets with emojis (used as "silver labels" for sentiment/emoji prediction) and tweets without emojis (for general NLP tasks).
Key Insights and Ideas
- Emojis as Universal Labels: The core premise is that emojis serve as a proxy for sentiment. A tweet containing a "❤️" is almost certainly positive, regardless of the language. This allows for "distant supervision," where models can be trained on millions of tweets without needing a human to manually label every single one.
- Cross-Lingual Semantic Preservation: By manually annotating a subset of 500 tweets per language, the researchers confirmed that the sentiment associated with specific emojis remains remarkably consistent across Hindi, Bengali, and Telugu. This suggests that emoji-based models could potentially be transferred across different linguistic communities with high accuracy.
- The "Lingua Franca" Barrier: The authors note that in India, English is often the language of administration and formal literature. This creates a paradox where languages have huge speaker bases but very little formal digital corpora, making social media (where people write as they speak) the best possible source for data.
Key Results
The study provides a quantitative look at how emojis correlate with human-perceived sentiment in these languages:
- High Sentiment Alignment: In Hindi, the "Smiling Face with Heart-Eyes" (😍) was manually verified as 99% positive. Conversely, the "Pouting Face" (😡) was 97% negative.
- Neutral Emojis: Some emojis, like the "Flushed Face" (😳) or "Thinking Face" (🤔), showed much higher neutrality (up to 98% neutral in Telugu), highlighting which emojis are useful for sentiment labels and which are better suited for nuanced emotion detection.
- Corpus Scale: The researchers successfully gathered 278,933 tweets specifically labeled with emojis across the three languages, providing a significant benchmark for future multilingual emoji prediction tasks.
Limitations and Future Work
- Code-Mixing: A major challenge in Indian social media is "Hinglish" or "Benglish"—mixing native languages with English. The current pipeline sometimes struggles to distinguish between a purely Hindi tweet and a heavily code-mixed one.
- Semantic Noise: Social media is messy. The authors noted instances where tweets were semantically non-compliant (e.g., a tweet about French fries that doesn't actually convey a clear sentiment), which can "poison" the training data for machine learning models.
- Temporal Decay: Twitter is a live platform. A snapshot taken today might not reflect the slang or trending topics of next year. The authors emphasize that these corpora need to be living documents that are updated regularly to remain useful.
Summary updated