TL;DR
This paper proposes a novel method to leverage code-switching (mixing languages within a single text) to identify "hope speech" – messages that diffuse hostility – in low-resource languages. By using code-switched documents as a bridge, they can effectively sample and identify hope speech in Romanized Hindi, a language with limited NLP resources, with minimal manual supervision.
What Problem Does This Paper Address?
Analyzing social media for geopolitical events and online discourse is crucial, but challenges arise when dealing with linguistically diverse user bases. A significant hurdle is code-switching, where users seamlessly blend multiple languages within a single post. While this phenomenon complicates downstream NLP tasks, this paper argues that it can be a valuable asset. Specifically, they address the challenge of identifying "hope speech" – messages promoting peace and diffusing hostility – in Romanized Hindi, a language widely used in India and Pakistan but lacking extensive NLP tools. Traditional approaches often focus on English-only content, leaving a significant portion of relevant discourse unanalyzed. This work aims to bridge this gap by using code-switched content as a means to access and analyze content in low-resource languages.
Methods
The core of the proposed method is a pipeline designed to sample code-mixed documents and then extract hope speech from the Hindi portion.
- Identifying Code-Mixed Documents: They use a "Code Mixing Index" (CMI) to quantify the extent of language mixing in a document. This index is calculated based on token-level language identification, leveraging a polyglot-embedding based method that requires minimal supervision. Documents with a CMI above a certain threshold are considered significantly code-mixed.
- Hope Speech Classification: A pre-existing hope speech classifier, trained on English comments, is applied to the identified code-mixed documents. This step identifies potential hope speech candidates within the mixed-language texts.
- Extracting Hindi Sub-parts: For the identified hope speech candidates, non-Hindi tokens are discarded to isolate the Hindi component. This creates a "seed set" of Hindi-centric hope speech.
- Nearest Neighbor Sampling: This Hindi seed set is then used to retrieve similar comments from the Romanized Hindi subset of the dataset. This is achieved by using comment embeddings and an active sampling technique that finds nearest neighbors in the embedding space.
- Manual Inspection: The final step involves manual annotation of the sampled comments to confirm they are indeed hope speech.
Datasets
- Primary Dataset (D): A corpus of 2.04 million comments from 2,890 YouTube videos related to the 2019 India-Pakistan conflict. This dataset is divided into English (Den) and Romanized Hindi (Dhe) subsets.
- Annotated Data Set (Dtrain): A subset of English comments used to train the initial hope speech classifier, consisting of 2,277 positive and 7,716 negative examples.
- Code-Mixed Subset (Dcm): Identified from Dhe U Den, containing comments with substantial code mixing.
- Hope Speech Subset (Dhope): Comments from Dcm predicted as hope speech by the classifier.
- Hindi Hope Speech Subset (Dhope_hindi): Hindi sub-parts extracted from Dhop.
Key Insights and Ideas
- Code-Switching as a Bridge: The paper's central idea is reframing code-switching from an impediment to a valuable tool for accessing content in low-resource languages.
- Human Well-being Focus: The work directly addresses the societal benefit of identifying and promoting positive, hostility-diffusing content in online discourse, particularly during sensitive geopolitical events.
- Minimal Supervision: The proposed pipeline requires significantly less manual annotation compared to traditional methods, making it more scalable and efficient.
- Leveraging Existing Resources: The method effectively utilizes a pre-trained English hope speech classifier to bootstrap analysis in a different language.
Key Results
- The proposed pipeline achieved a nearly 10-fold improvement in sampling hope speech from the Romanized Hindi subset (Dhe) compared to a random sampling baseline.
- The best performing sampling method (NN-Sample(Dhope_hindi)) achieved an accuracy of 31.68%, requiring minimal manual supervision only in the final inspection step.
- The polyglot-embedding based language identification method demonstrated high accuracy in token-level language identification, even on noisy social media text.
- Extracting the Hindi sub-part of code-mixed comments was shown to be crucial for sampling from the low CMI (pure Hindi) region of Dhe, leading to less code-mixed results.
Limitations and Future Work
- Noisy Approximations: The authors acknowledge that their pipeline relies on noisy approximations in estimating CMI, extracting Hindi sub-parts, and detecting hope speech.
- Manual Inspection Still Needed: While significantly reduced, manual inspection is still required in the final step to confirm sampled comments as hope speech.
- Future Work: The authors suggest that introducing more supervision (e.g., expanding the manually annotated hope speech set) could further improve performance. They also highlight the potential for plugging in better-performing algorithms at various stages of the pipeline without disrupting the overall flow, suggesting avenues for future performance enhancements.
Summary updated