The D-WISE Tool Suite: Multi-Modal Machine-Learning-Powered Tools Supporting and Enhancing Digital Discourse Analysis

Florian Schneider, Tim Fischer, Fynn Petersen-Frey, Isabel Eiser, Gertraud Koch, Chris BiemannACL 2023

TL;DR

The D-WISE Tool Suite (DWTS) is a novel, open-source platform designed to revolutionize digital discourse analysis in the humanities. By integrating state-of-the-art machine learning from NLP and computer vision, DWTS empowers researchers to analyze vast, heterogeneous, and multi-modal datasets (text, image, audio, video) that were previously unmanageable, offering a powerful alternative to proprietary and limited existing tools.

What Problem Does This Paper Address?

In today's digital age, societies generate an overwhelming amount of data in diverse formats – news articles, social media posts, videos, podcasts, and more. This "digital deluge" presents a significant challenge for Digital Humanities (DH) researchers conducting qualitative discourse analysis. Manually sifting through this heterogeneous, multi-modal data to uncover complex sociological patterns and discussions is practically impossible for individuals or even research groups. While existing Computer-Assisted Qualitative Data Analysis Software (CAQDAS) and annotation tools exist, they often fall short. Many are proprietary, lack support for multi-modal data, or fail to leverage advanced machine learning techniques for automatic annotation and sophisticated search capabilities. This leaves a critical gap in the ability of DH researchers to effectively analyze the rich discourses embedded within contemporary digital content.

Methods

The D-WISE Tool Suite (DWTS) tackles this problem by building a comprehensive, interdisciplinary platform that combines cutting-edge machine learning with user-centric design. The core of DWTS lies in its multi-modal preprocessing pipeline, which leverages recent advances in Natural Language Processing (NLP) and Computer Vision. This pipeline automates tasks such as:

  • Named Entity Recognition (NER) for text.
  • Multi-modal similarity search that allows finding related content across different modalities (e.g., finding images related to a text passage).
  • Object detection in images and videos.
  • Image captioning to generate textual descriptions of images.
  • Automatic Speech Recognition (ASR) for transcribing audio and video.
  • Video understanding tasks.

DWTS also incorporates advanced search functionalities, including lexical and semantic similarity searches, and supports manual annotations for text, audio, video, and image regions. A key aspect is its extensible and scalable open-source architecture, built on modern technologies like Python, React, PostgreSQL, and Docker, making it accessible and deployable for research groups.

Datasets

The paper does not explicitly list specific datasets used for training or evaluation. Instead, it emphasizes the system's ability to handle "heterogeneous, unstructured, and multi-modal data" in general. The system is designed to ingest various formats including HTML, PDF, DOCX, raw text files, images, audio, and video. The authors also mention an external crawler using Scrapy and Beautifulsoup for scraping websites, implying a broad data acquisition capability.

Key Insights and Ideas

  • Bridging the Gap: DWTS democratizes access to powerful ML tools for DH researchers, who may not have deep technical expertise in ML themselves.
  • True Multi-modality: The system's strength lies in its ability to process and link information across text, image, audio, and video, reflecting the complex nature of real-world discourse.
  • Interdisciplinary Design: The co-creation between cultural anthropologists and computer scientists ensures the tool is not just technically advanced but also practically useful and aligned with the needs of qualitative discourse analysis.
  • Scalability and Openness: The open-source, Docker-based deployment makes DWTS accessible, customizable, and scalable, addressing the limitations of proprietary software and complex installation processes.
  • Integrated Workflow: DWTS supports the entire research process, from data import and preprocessing to exploration, annotation, coding, and documentation, streamlining the analytical workflow.

Key Results

The paper focuses on demonstrating the functionality and architecture of the DWTS rather than presenting quantitative performance metrics on specific tasks. The "results" are primarily qualitative:

  • The system successfully integrates various ML models for automatic annotation and enrichment of multi-modal data.
  • It provides a user-friendly interface for exploring, searching, and annotating diverse document types.
  • The multi-modal similarity search is a key feature enabling novel analytical pathways.
  • The system supports collaborative research through user roles and shared project spaces.
  • The integrated logbook and memo features enhance the documentation and reflection aspects of qualitative research.

Limitations and Future Work

The authors acknowledge several limitations:

  • ML Model Limitations: ML models are not error-free and can contain biases. DWTS attempts to mitigate this by visualizing confidence scores and offering traditional, ML-free alternatives like lexical search. However, errors in preprocessing or entity recognition can still lead to biased results.
  • System Design Bias: The envisioned workflow and system design might inadvertently impose constraints on user workflows.
  • Privacy and Security: While DWTS can be self-hosted, users upload potentially sensitive data. The opt-in logging feature aims to give users control over their usage data.
  • Ongoing Development: Features like similarity search for raw audio and video documents, and 4-way multi-modal similarity search (text, image, audio, video) are noted as work-in-progress and not yet fully released in the demonstration instance.

Future work will likely focus on refining these ongoing developments and continuing to engage with domain experts to address limitations related to ML, privacy, and bias.

Summary updated

BibTeX
@inproceedings{schneider2023d,
  author = {Florian Schneider and Tim Fischer and Fynn Petersen-Frey and Isabel Eiser and Gertraud Koch and Chris Biemann},
  title = {{The D-WISE Tool Suite: Multi-Modal Machine-Learning-Powered Tools Supporting and Enhancing Digital Discourse Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure