Aggregating distributed STT, MT, and information extraction engines: the GALE interoperability-demo system

John F. Pitrelli, Burn L. Lewis, Edward A. Epstein, Martin Franz, Daniel Kiecza, Jerome L. Quinn, Ganesh Ramaswamy, Amit Srivastava, Paola VirgaInterspeech 2008

TL;DR

This paper introduces the GALE Interoperability-Demo (IOD) system, a groundbreaking demonstration of how to seamlessly integrate diverse natural language processing (NLP) engines (speech-to-text, machine translation, information extraction) into a single, functional pipeline. It showcases a practical approach to building complex NLP applications by overcoming the significant challenges of inter-engine communication, data format mismatches, and distributed computing.

What Problem Does This Paper Address?

As individual NLP technologies like speech recognition (STT) and machine translation (MT) mature, the next frontier is combining them to tackle more ambitious tasks. However, building systems that effectively orchestrate multiple, independently developed NLP engines is fraught with challenges. These include:

  • Interoperability: Engines often have incompatible input/output formats and data structures.
  • Heterogeneity: Engines are built using different programming languages and run on diverse operating systems.
  • Distribution: Engines are often developed by different teams and reside on separate computing platforms, necessitating remote communication.
  • Fault Tolerance: In a complex system, individual engine failures or network issues can bring the entire system down.

The GALE IOD system aims to address these challenges by demonstrating a robust framework for aggregating these disparate engines to process Arabic broadcast news into browsable English text.

Methods

The core of the GALE IOD system's solution lies in the Unstructured Information Management Architecture (UIMA). UIMA provides a standardized framework for interoperation, enabling:

  • Common Analysis Structure (CAS): A central data structure that holds all information about a segment of data, including various "views" (e.g., audio, Arabic text, English text) and annotations.
  • Type System: A defined set of data types (like AudioToken for words) that allows engines to share information consistently.
  • Data Reorganization Modules: Specialized UIMA components that transform data from one view or format to another, bridging the gap between engines. For example, converting STT output into an Arabic text view for MT.
  • UIMA-AS (Aggregate Services): A component that facilitates communication with remotely located engines, allowing them to be deployed as services.
  • Context-Dependent Processing: UIMA's ability to manage complex data flows, including splitting, merging, and buffering segments, which is crucial for engines requiring contextual information (like story boundary detection).
  • Fault Tolerance Mechanisms: The system is designed to handle engine failures gracefully, allowing for the removal of failed engines and continued operation, especially when multiple engines perform the same function.

Datasets

The paper describes the system processing four hours of Arabic news daily from two Arabic news networks, Al-Arabiya and Al-Jazeera, for over a year. This represents a continuous, real-world data stream rather than a static, pre-defined benchmark dataset. The system's output is made browsable via a web interface.

Key Insights and Ideas

  • UIMA as a Unifier: UIMA proves to be a powerful abstraction layer that effectively hides the underlying heterogeneity of engines, enabling their seamless integration.
  • The Power of Aggregation: Even with imperfect individual engines, combining them can yield surprisingly useful and functional results, opening doors for new applications.
  • Data Reorganization is Crucial: The success of the system hinges on intelligent data transformation modules that adapt engine outputs to the inputs required by subsequent engines.
  • Fault Tolerance is Non-Negotiable: For any practical, large-scale NLP system, robust error handling and fault tolerance are paramount. The ability to dynamically remove failed engines is a key enabler.
  • Focus on Interoperability Over Tight Coupling: The paper emphasizes that while some engines might be designed for tight integration, the goal of IOD is to achieve functionality through broad interoperability, imposing minimal constraints on individual engines.

Key Results

The GALE IOD system successfully demonstrates the aggregation of 12 engines, including:

  • Speech-to-text (STT)
  • Entity detection (ED)
  • Machine translation (MT)
  • Multi-engine MT (MEMT)
  • Story-boundary detection
  • Topic clustering
  • Topic summarization
  • Headline generation

The output is a browsable English text representation of Arabic news videos, organized by topic, with clickable headlines leading to summaries and story details. The system has been running continuously for over a year, processing significant amounts of real-world news data.

Limitations and Future Work

  • Performance Evaluation Complexity: The authors acknowledge that quantitatively evaluating the accuracy of such a complex, multi-engine system is challenging due to the difficulty in obtaining ground-truth data and defining objective metrics for all components. They are developing a formal framework for this.
  • Engine Constraints: Running engines within an aggregate can impose constraints not present in isolation, such as handling empty strings for MT or dealing with special codes like "unk" from STT. The system requires engines to adapt to these new requirements.
  • Future Work: The paper suggests future expansion will include adding engines for language identification, speaker identification, and text-to-speech synthesis. They also propose a "forgetting" mechanism for history-maintaining engines to manage old topics.

Summary updated

BibTeX
@inproceedings{pitrelli2008aggregating,
  author = {John F. Pitrelli and Burn L. Lewis and Edward A. Epstein and Martin Franz and Daniel Kiecza and Jerome L. Quinn and Ganesh Ramaswamy and Amit Srivastava and Paola Virga},
  title = {{Aggregating distributed STT, MT, and information extraction engines: the GALE interoperability-demo system}},
  booktitle = {Interspeech},
  year = {2008},
}
Figure