Universität Karlsruhe (TH) at TRECVID 2007

Hazim Kemal Ekenel, Mika Fischer, Hua Gao, K. Kilgour, J. S. Marcos, Rainer StiefelhagenTRECVID 2007 · OpenReview

This paper summarizes the debut participation of the interACT Research Center at Universität Karlsruhe (TH) in the TRECVID 2007 evaluation. The researchers introduced a modular software framework designed to address two core tasks in video analysis: Shot Boundary Detection (SBD) and High-Level Feature Detection (HLFD).

The Challenge of Automated Video Indexing

As digital video libraries expand, automated indexing becomes essential for efficient retrieval. This process begins with Shot Boundary Detection, segmenting video into continuous camera actions. Subsequently, High-Level Feature Detection "tags" these segments with semantic concepts such as "Face," "Car," or "Mountain." The primary difficulty lies in the diversity of video content; transitions can be abrupt (cuts) or gradual (dissolves), and semantic identification requires synthesizing disparate data types including color, texture, shape, and spoken dialogue.

Methodology

The authors adopted a "divide and conquer" strategy, implementing specialized systems for segmentation and classification.

Shot Boundary Detection (SBD)

The SBD system utilizes a modular pipeline where specific detectors target different transition types:

  • Cuts: Identified using RGB histogram differences and kurtosis within a seven-frame window.
  • Fast Dissolves: Modeled via a least-squares approach to determine if a frame sequence represents a linear transition.
  • Fades: Triggered by monochrome frames with low intensity standard deviation.
  • Fusion: A final module resolves conflicts between overlapping detector claims.

High-Level Feature Detection (HLFD)

The HLFD system follows a three-stage process: extraction, classification, and fusion.

High-level feature detection architecture, illustrating the flow from feature extraction to multi-modal fusion.
  1. Extraction: Descriptors are pulled for Color (histograms, moments), Texture (Wavelets), Shape (Geometric Blur), and Text (Relative Term Frequency from speech transcripts).
  2. Classification: Each feature set is processed by a dedicated Support Vector Machine (SVM) to generate confidence scores for 39 categories.
  3. Fusion: Scores are combined using methods ranging from simple voting to complex neural networks and inter-category correlation matrices.

Key Insights

  • Efficiency through Modularity: By allowing modules to share pre-computed features, the framework minimizes redundant calculations.
  • The "Text Paradox": Integrating Automatic Speech Recognition (ASR) data did not significantly boost performance. Descriptive words often do not coincide temporally with the visual representation of the concept.
  • Semantic Context: Utilizing a correlation matrix allows the system to leverage relationships between categories (e.g., the presence of "Water" increasing the likelihood of "Boat").

Results and Performance

The system demonstrated strong performance in specific areas while revealing clear limitations in others.

SBD performance metrics across various experimental runs, highlighting the disparity between cut and gradual transition accuracy.

In SBD tasks, the system was highly effective at detecting hard cuts, achieving recall and precision rates between 93% and 94%. However, gradual transitions (dissolves) proved challenging, with precision dropping as low as 4.4% in some configurations due to high false-positive rates. In HLFD, the best-performing run (UKA3) achieved a Mean Inferred Average Precision (InfAP) of 2.3%, with simple fusion methods often outperforming more complex neural network approaches.

Conclusion

The interACT Research Center's debut highlighted the strengths of modular video analysis while identifying critical areas for future work. Key improvements will focus on tuning SBD thresholds for varied lighting conditions, transitioning to SVM-based dissolve detection, and optimizing the weighting of visual versus textual cues for semantic tagging.

Summary updated

BibTeX
@inproceedings{ekenel2007universitat,
  author = {Hazim Kemal Ekenel and Mika Fischer and Hua Gao and K. Kilgour and J. S. Marcos and Rainer Stiefelhagen},
  title = {{Universität Karlsruhe (TH) at TRECVID 2007}},
  booktitle = {TRECVID},
  year = {2007},
  url = {https://openreview.net/forum?id=66QYXdhhU-},
}
Figure