TL;DR
This paper introduces a software tool that unifies over 40 diverse toxic comment classification datasets into a single, easily accessible format. This addresses the significant challenge of fragmented and heterogeneous data in the field, enabling more robust research and practical applications by simplifying data retrieval, standardization, and mapping of class labels.
What Problem Does This Paper Address?
The field of toxic comment classification has seen a surge in research, leading to the release of numerous annotated datasets. However, these datasets are often scattered across different platforms, come in various formats, and use inconsistent labeling schemes. This fragmentation makes it difficult for researchers to effectively train and evaluate models, often forcing them to rely on single, potentially biased datasets. While existing overviews of datasets exist, they don't provide a unified, programmatic way to access and use this data, hindering progress in developing generalized and robust toxic comment detection systems.
Methods
The core contribution of this paper is a software tool that automates the process of data integration. The tool performs three key steps:
- Retrieval: It automatically downloads datasets from various sources like GitHub repositories, university web pages, and cloud storage platforms.
- Standardization: It converts all datasets into a unified CSV format, handling diverse file types (CSV, Excel, SQL dumps, TXT files) and column separators.
- Label Mapping: It addresses the challenge of inconsistent class labels by providing a configurable mapping system. This allows users to unify labels with similar meanings across different datasets and to create standardized label sets, such as a binary "toxic" vs. "non-toxic" classification.
The tool is available as both a GitHub repository and a PyPI package, allowing easy installation and integration into Python workflows.
Datasets
The paper presents a collection of over 40 toxic comment datasets. These datasets originate from various sources, including:
- Platforms: Twitter, Facebook, Wikipedia, forums, and news articles.
- Languages: Thirteen different languages are covered, including Arabic, Danish, English, French, German, Greek, Hindi, Indonesian, Italian, Marathi, Portuguese, Slovenian, and Turkish.
- Labeling Schemes: The datasets encompass a wide range of toxicity-related labels, such as sexism, aggression, hate speech, offense, and abuse.
The paper provides a detailed table (Table 1) listing each dataset, its size, source, language, and class labels. It notes that the collection is continuously growing, with plans to integrate more datasets.
Key Insights and Ideas
- Data Fragmentation is a Major Bottleneck: The paper highlights that despite the growing research in toxic comment classification, the lack of unified and accessible datasets significantly hinders progress.
- Label Inconsistency is More Than Just Different Names: The paper emphasizes that labels can represent different granularities of toxicity or even be specific to certain contexts (e.g., "female football players"), making direct comparison and aggregation challenging.
- Focused Data Collection Can Lead to Bias: The authors point out that datasets collected with a specific focus on toxic comments (e.g., using keyword filtering) can create biased samples that are not representative of real-world online discourse, potentially leading to models that overfit to specific keywords.
- Re-hydration of Twitter Data: The paper mentions the common practice of collecting tweet IDs and then re-hydrating them via the Twitter API, a process their tool automates.
Key Results
The paper's primary result is the creation and release of the toxic-comment-collection software tool. This tool provides:
- Easy Access: A unified API for accessing a large number of toxic comment datasets.
- Standardized Format: All datasets are presented in a consistent format, simplifying downstream processing.
- Metadata Filtering: Users can filter datasets based on language, platform, and class labels.
- Comprehensive Collection: The current collection includes data from 13 languages and 12 platforms, with 162 distinct class labels before mapping.
- Large Scale: The unified collection contains over 812,993 samples.
- Binary Classification Option: The tool facilitates creating a binary "toxic" vs. "non-toxic" classification by mapping various labels.
Limitations and Future Work
Limitations:
- Dataset Availability: Some datasets are not publicly accessible and require special requests, which the tool aims to address by providing contact information.
- Class Imbalance: The paper acknowledges that many individual datasets suffer from inherent class imbalance, which is a general challenge in this domain.
- Bias in Focused Datasets: As discussed in the "Key Insights" section, datasets collected with specific filtering can introduce biases.
Future Work:
- Continuous Dataset Integration: The authors plan to continuously add more datasets to the collection.
- Contact Information for Inaccessible Datasets: Providing contact details for datasets that are not publicly available but can be obtained upon request.
- Fostering Research: The tool is intended to foster research on toxic comments, multilingual classification, and the development of robust systems for practical applications.
Summary updated