Introduction
In the domain of youth online safety, the timing of an intervention is as critical as its accuracy. Current machine learning models for detecting cyberbullying or sexual solicitation often face a "Goldilocks" problem: single-message analysis is fast but lacks the context to distinguish a joke from a threat, while full-conversation analysis provides high accuracy but is often "after-the-fact." By the time a full conversation is flagged, the harm has frequently already occurred. This research addresses this trade-off by proposing a dynamic triage system that identifies the "optimal stopping point" to intervene in high-risk interactions.
Methodology: The Two-Level RL Framework
The researchers developed a novel two-level algorithmic framework designed to balance detection speed with predictive performance.
- Level 1: The RL Agent (DQN): A Deep Q-Network (DQN) acts as a gatekeeper. As messages flow in, the agent chooses between two actions: continue reading for more context or stop and forward the messages for priority evaluation. The agent is trained using a reward function that balances the cost of false alarms against the penalty of missing high-risk interactions.
- Level 2: Priority Classification: Once the RL agent triggers a "stop," the sequence is passed to a deep learning classifier. The study evaluated several architectures, including LSTM, Bi-LSTM, CNN, and DNN, utilizing ELECTRA embeddings to categorize conversations as high or low priority.
Dataset and Experimental Setup
The study utilized the Instagram Data Donation (IGDD) dataset, which includes 23,000 private conversations and over 7 million messages donated by 194 youth (ages 13-21). This dataset is particularly significant because it captures private, real-world interactions with ground-truth labels provided by the youth themselves, who categorized their own experiences as safe, low-risk, or high-risk.
Key Results
The framework demonstrated that models do not need a complete conversation to identify significant risks. The RL-driven approach successfully identified "junctures" where risk became apparent, allowing for much faster triage than traditional methods.
- Massive Speedup: The time required to identify high-risk conversations dropped from an average of 126 minutes (baseline) to just 16 minutes using the RL framework.
- Maintained Accuracy: While waiting for a full conversation yielded an accuracy of approximately 89%, the RL-triggered Bi-LSTM (DQN) model achieved 84% accuracy. In a safety context, this minor drop in accuracy is a small price to pay for an intervention that occurs nearly two hours earlier.
- Computational Efficiency: Simulations showed that the high-priority triage system processed high-risk conversation batches significantly faster than systems treating all conversations equally, proving its scalability for large platforms.
Conclusion and Future Work
By treating risk detection as a sequential decision-making problem, this framework allows social media platforms to direct limited human moderation resources to the most dangerous situations first. Future work will look to incorporate non-textual features such as images and metadata, and explore how to best integrate this triage system into real-world human-in-the-loop moderator workflows.
Summary updated