Video Retrieval System Using Automatic Speech Recognition for the Japanese Diet

Mikitaka Masuyama, Tatsuya Kawahara, Kenjiro MatsudaCOLING 2024

TL;DR

This paper introduces a video retrieval system for Japanese parliamentary proceedings that leverages automatic speech recognition (ASR) to create searchable video archives. By accurately transcribing speeches and linking them to timestamps, the system allows users to pinpoint and play specific segments of debates via keyword search, significantly enhancing access to and analysis of parliamentary discourse.

What Problem Does This Paper Address?

Analyzing parliamentary proceedings has traditionally relied on textual minutes, which, while useful, lack the richness of visual and auditory information. The increasing availability of video streams presents a challenge: how to efficiently navigate and search hours of footage to find specific moments or discussions. Traditional methods of searching video archives are often limited to metadata like date or title, making it difficult to locate precise utterances by specific speakers or on particular topics. This paper addresses the need for a system that can bridge the gap between raw video data and searchable textual information, enabling deeper analysis of parliamentary debates.

Methods

The core of the system is a two-part approach:

  1. Automatic Speech Recognition (ASR) for Timestamping: The system utilizes a specialized ASR model trained on a large corpus of Japanese parliamentary speech. This model achieves high accuracy (up to 95%) and generates timestamps for each transcribed word. This is crucial for linking spoken content to specific points in the video. The training method employs "lightly-supervised training based on statistical language model transformation," which reduces the need for extensive manually transcribed data.
  2. Video Retrieval System: This subsystem integrates the ASR output with the Diet Library's proceedings database and the Diet secretariats' video libraries. Users can perform keyword searches, which are automatically translated into Japanese if entered in English. The system then retrieves relevant video clips corresponding to the search terms and their associated timestamps. The interface allows for partial playback of these clips, highlighting the searched speech segment.

Additionally, the system has been extended to:

  • Extract Supplementary Materials: Using speech and pattern recognition, it can identify and extract video clips focusing on supplementary materials (like figures and tables) presented during debates, making their content searchable.
  • Analyze Non-Verbal Communication: The authors are developing capabilities to automatically extract and analyze facial expressions and body language, though this is still under development.

Datasets

The paper mentions the use of:

  • Japanese parliamentary speech data: Used for training the ASR system. This data is described as "large amount" and covers terms and expressions specific to the Parliament.
  • Diet Library's proceedings database: Contains digitized minutes of parliamentary meetings.
  • Diet secretariats' deliberation video libraries: Archives of parliamentary meeting videos.

The system itself acts as a benchmark for searching these combined datasets. The paper notes that the database covers meetings from January 2010 for the House of Representatives and December 2012 for the House of Councillors.

Key Insights and Ideas

  • Bridging the Text-Video Divide: The fundamental insight is that accurate ASR with precise timestamping is the key to unlocking the analytical potential of parliamentary video archives, making them as searchable as text documents.
  • Beyond Simple Transcription: The system goes beyond basic transcription by enabling direct retrieval of video segments, allowing for a more nuanced understanding of debates than text-only minutes can provide.
  • Facilitating Deeper Analysis: By allowing users to pinpoint specific utterances and view accompanying visual cues (like supplementary materials or speaker expressions), the system supports richer qualitative analysis of political discourse.
  • Potential for Broader Applications: The techniques are adaptable to other parliamentary bodies, local government meetings, international conferences, and even other forms of video content where precise content retrieval is valuable.

Key Results

  • High ASR Accuracy: The ASR system consistently achieves character accuracy of over 90%, improving to 95% at the time of writing.
  • Searchable Video Archives: The system successfully enables keyword searching of parliamentary video segments, allowing users to directly retrieve and play relevant portions of debates.
  • Extended Coverage: The system covers plenary and committee meetings from January 2010 (House of Representatives) and December 2012 (House of Councillors).
  • Supplementary Material Search: The system can now search for video portions focusing on supplementary materials by their content.

Limitations and Future Work

Limitations: * Supplementary Material Search: While the system can search for supplementary materials, the initial description of the ASR output notes it "may contain irrelevant fillers and incorrectly recognized errors." The minutes are "official" by correcting these, implying the raw ASR output might still require some post-processing for perfect accuracy. * Non-Verbal Communication Analysis: The analysis of facial expressions and body language is still under development. * English Interface Limitations: The English interface for searching deliberation videos is less comprehensive than the Japanese one, offering fewer search options.

Future Work: * Extension to Local Governments: The system is being extended to cover meetings of local governments to investigate dialectal linguistic variations. * Multi-linguistic Support: The authors suggest developing multi-linguistic speech recognition for parliaments that operate in multiple languages. * International Conferences: The system's potential is highlighted for searching proceedings of international conferences that are increasingly relying on video dissemination. * Improved Non-Verbal Analysis: Continued development of tools to automatically extract and analyze speaker's facial expressions and body language.

Summary updated

BibTeX
@inproceedings{masuyama2024video,
  author = {Mikitaka Masuyama and Tatsuya Kawahara and Kenjiro Matsuda},
  title = {{Video Retrieval System Using Automatic Speech Recognition for the Japanese Diet}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure