Name-It: Naming and Detecting Faces in Video by the Integration of Image and Natural Language Processing

TL;DR

This paper introduces "Name-It," a system that automatically associates faces with names in news videos by integrating advanced image and natural language processing. By leveraging improved face tracking and sophisticated name extraction techniques, Name-It significantly enhances the ability to access and index multimedia content, paving the way for more intelligent video analysis.

What Problem Does This Paper Address?

The proliferation of multimedia content, particularly news videos, presents a significant challenge in accessing and understanding its underlying information. While videos contain rich visual and textual data, extracting meaningful "content" – such as identifying who is speaking or appearing on screen and associating them with their names – remains difficult. Existing systems often struggle with accurate face detection and robust name extraction, especially when faces are not perfectly frontal or when names are not explicitly stated in close proximity to the visual appearance. This paper aims to bridge this gap by developing a system that can reliably link visual representations of people (faces) with their corresponding textual identifiers (names) within news broadcasts.

Methods

Name-It employs a two-pronged approach, integrating sophisticated image and natural language processing techniques:

  • Image Processing (Face Extraction and Identification):
    • Face Tracking: Instead of relying solely on scene change detection, Name-It incorporates face tracking. This allows for the extraction of face sequences, providing crucial information about face duration and enabling the selection of the most frontal view of a face within a sequence.
  • Frontal Face Selection: A novel method is used to identify the "most frontal" face from a sequence. This involves clustering skin regions and calculating a "frontal factor" based on eye positions and face width, ensuring that the most recognizable face is used for identification.
  • Eigenface-Based Face Identification: Standard eigenface techniques are used to identify faces by comparing them in a high-dimensional eigenface space.
  • Natural Language Processing (Name Extraction):
    • Lexical/Grammatical Analysis: The system goes beyond simple dictionary lookups for proper nouns. It employs advanced lexical and grammatical analysis, utilizing dictionaries, thesauri (like WordNet), and parsers to extract names and identify potential persons of interest.
  • Topic Structure Knowledge: Name-It leverages the inherent structure of news transcripts. It identifies key indicators (like ">>>" for topics and "»" for paragraphs) to understand the flow of information.
  • Scoring System: A multi-faceted scoring system is used to rank name candidates. This includes:
  • Grammatical Score: Based on whether a word is a proper noun and its grammatical role.
  • Lexical Score: Based on whether the word refers to a person, social group, or other entity.
  • Situational Score: Based on the verb associated with the word (e.g., speech, attendance).
  • Positional Score: Crucially, this score considers the word's position within the topic and its proximity to live video segments, recognizing that anchors often introduce speakers before they appear on screen.
  • Face-Name Association:
    • Co-occurrence Factor: The core of the association is a "co-occurrence factor" that measures how often a face and a name appear together. This factor is extended to incorporate face duration and the scores of name candidates.
  • Normalization: The co-occurrence factor is normalized to mitigate the "anchor person problem," where anchors might be associated with many names due to their frequent appearance.

Datasets

The system was evaluated using CNN Headline News videos. The paper does not mention the creation of new datasets but rather the application of the system to existing news broadcast data.

Key Insights and Ideas

  • Integration is Key: The paper strongly emphasizes that combining advanced image and natural language processing is crucial for effectively handling multimedia content, as neither technique alone is sufficient.
  • Face Tracking for Better Identification: Moving beyond static frame analysis to face tracking provides richer information (like duration) that significantly improves face identification accuracy.
  • Contextual Name Extraction: Simply finding names in text is insufficient. Understanding the structure of news topics and the situational context of words (e.g., who is speaking, when) is vital for accurate name candidate generation.
  • The "Anchor Person Problem": The authors explicitly address and propose a solution for the challenge of anchors appearing frequently and being incorrectly associated with numerous names.
  • Positional Clues Matter: The timing and placement of names within a news transcript, especially their relation to live video segments, provide strong clues for accurate face-name association.

Key Results

The paper demonstrates the effectiveness of the enhanced Name-It system through experimental results:

  • Improved Face Extraction: The new face extraction method, incorporating tracking and frontal face selection, successfully detected faces in scenarios where the former system failed (e.g., due to person looking down or scene changes). It also produced more frontal and thus more identifiable faces.
  • Enhanced Name Candidate Retrieval: Experiments showed that the improved name extraction significantly outperformed the previous version. For example, when retrieving names for a face, the new system correctly identified "GINGRICH" as the second candidate, whereas the old system failed to place it within the top thirty. Similarly, when retrieving faces for the name "CLINTON," the new system achieved a much higher accuracy rate for the top candidates.
  • Efficiency: Processing a 30-minute news video took approximately 1.5 hours on an SGI workstation, with most of this time spent on parsing.

Limitations and Future Work

  • Limitations:
    • The system relies on closed-caption texts as transcripts, which might not always be perfectly synchronized or complete.
  • The "anchor person problem" is addressed but might still pose challenges in complex scenarios.
  • The computational cost of processing remains significant, though improved over previous versions.
  • Future Work:
    • The authors suggest that the system's capabilities can be extended to applications like automated video indexing by person's name and the automated creation of large face-name correspondence databases.
  • Further refinement of both image and natural language processing components is implied for even greater accuracy and efficiency.

Summary updated

BibTeX
@inproceedings{1997name,
  title = {{Name-It: Naming and Detecting Faces in Video by the Integration of Image and Natural Language Processing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1997},
}
Figure