Interactive Search and Exploration in Online Discussion Forums Using Multimodal Embeddings

Iva Gornishka, Stevan Rudinac, Marcel WorringarXiv 2019 · OpenReview

Abstract

In this paper, we present ISOLDE, a novel interactive multimodal learning system designed to facilitate search and exploration within large networks of social multimedia users. The system allows analysts to identify users of interest and discover similar actors through an interactive learning framework. Our approach leverages multimodal representations of users, words, and concepts, learned simultaneously using a general-purpose neural embedding model. We demonstrate that these representations are effective for categorizing users and automatically generating community profiles. The system's utility is evaluated using artificial actors in relevance feedback scenarios across datasets from the Stormfront forum and Twitter, highlighting the importance of multimodal context in high-semantic online discussions.

Introduction: The Needle in the Social Media Haystack

Social scientists and law enforcement agencies increasingly face a daunting challenge: navigating the vast, heterogeneous data generated by online discussion forums and social media. These platforms are not only massive in scale but also complex, containing a mix of text, images, and intricate user-to-user metadata.

Existing analytical tools often suffer from three primary flaws: they are static, providing only snapshots; they are unimodal, typically ignoring visual context; and they are inflexible, struggling to adapt as the vocabulary of extremist or niche communities evolves. ISOLDE (Interactive Search and Exploration in Online Discussion Forums) was developed to bridge this gap by prioritizing interactivity, scalability, and comprehensibility.

System Architecture and Methodology

The ISOLDE system is built upon a three-pillared architecture designed to turn raw social data into actionable insights.

Overview of the ISOLDE system architecture, illustrating the pipeline from data collection to the interactive user session.

1. Multimodal Neural Embeddings

At the core of the system is the StarSpace neural embedding model. Unlike traditional models that treat text and images as separate silos, StarSpace maps users, words, and visual concepts into a single high-dimensional space. This unified mapping allows the system to recognize semantic relationships across modalities—for instance, linking a specific extremist slogan (text) to a particular symbol (visual concept).

2. Automated User Profiling

To prevent analyst burnout, ISOLDE generates concise user "summaries." Instead of requiring an expert to read thousands of posts, the system uses an algorithm to select the most representative and diverse content (words and images) from a user's history. This selection is based on the user's specific position within the embedding space, ensuring the profile captures the essence of their online persona.

3. Interactive Relevance Feedback

ISOLDE employs a "human-in-the-loop" approach. During a session, an analyst marks a small subset of users as "relevant" or "irrelevant." A linear Support Vector Machine (SVM) then re-ranks the entire dataset in real-time, surfacing similar users based on the analyst's specific criteria.

The ISOLDE user interface, showing the interactive visualization of user clusters and the relevance feedback panel.

Experimental Evaluation

The researchers evaluated ISOLDE on two datasets known for violent online extremism:

  • Stormfront: Approximately 2 million posts from a white nationalist forum.
  • Twitter: 2.7 million messages from various extremist-leaning groups, including ISIS supporters and far-right activists.

Key Insights

  • Context Over Keywords: While keyword-based methods (like TF-IDF) are effective for narrow topics, neural embeddings excel in abstract categories where sentiment and context are paramount. In the "Stormfront Ladies" section, for example, the system identified communities based on the nature of the discussion rather than just recurring terms.
  • Visual Concepts as Universal Language: In multilingual environments, such as the French-speaking sections of Stormfront, visual concepts acted as a language-independent bridge, allowing the model to maintain accuracy where text-only models failed.
  • Performance Gains: Using simulated "artificial actors" to provide feedback, the researchers observed significant increases in Mean Average Precision (MAP) over ten rounds of interaction, proving the system's ability to learn and adapt to an analyst's needs in real-time.

Conclusion and Future Directions

ISOLDE demonstrates that multimodal embeddings combined with interactive feedback can significantly enhance the exploration of complex social networks. While the current prototype relies on static modality weighting, future iterations could feature dynamic weighting to automatically prioritize the most informative modality for a given task. Additionally, further work is needed to make these high-dimensional embeddings more intuitive for non-technical domain experts, ensuring that the "black box" of neural networks becomes a transparent tool for social analysis.

Summary updated

Abstract

In this paper we present a novel interactive multimodal learning system, which facilitates search and exploration in large networks of social multimedia users. It allows the analyst to identify and select users of interest, and to find similar users in an interactive learning setting. Our approach is based on novel multimodal representations of users, words and concepts, which we simultaneously learn by deploying a general-purpose neural embedding model. We show these representations to be useful not only for categorizing users, but also for automatically generating user and community profiles. Inspired by traditional summarization approaches, we create the profiles by selecting diverse and representative content from all available modalities, i.e. the text, image and user modality. The usefulness of the approach is evaluated using artificial actors, which simulate user behavior in a relevance feedback scenario. Multiple experiments were conducted in order to evaluate the quality of our multimodal representations, to compare different embedding strategies, and to determine the importance of different modalities. We demonstrate the capabilities of the proposed approach on two different multimedia collections originating from the violent online extremism forum Stormfront and the microblogging platform Twitter, which are particularly interesting due to the high semantic level of the discussions they feature.
BibTeX
@misc{gornishka2019interactive,
  author = {Iva Gornishka and Stevan Rudinac and Marcel Worring},
  title = {{Interactive Search and Exploration in Online Discussion Forums Using Multimodal Embeddings}},
  year = {2019},
  eprint = {1905.02430},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.02430},
}
Figure