TL;DR
This paper introduces a flexible "Event-Level Feature" (ELF) learning framework to effectively classify consumer events (like weddings or birthdays) from unconstrained photo and video collections. By combining cross-domain learning (leveraging external datasets) and within-domain learning, along with both image-level and region-level features, they achieve significant improvements in event classification accuracy.
What Problem Does This Paper Address?
Organizing and understanding personal photo and video collections is a major challenge. Automatic albuming systems group media into "events" based on time and visual similarity, but these events can be noisy, contain mixed media (photos and videos), and exhibit high visual diversity even within the same semantic category. Furthermore, these automatically generated events lack explicit labels, making direct classification difficult. The paper aims to bridge this gap by developing a robust semantic event classification system that can handle these real-world complexities, moving beyond simple image-level analysis to understand the content of entire events.
Methods
The core contribution is a two-step Event-Level Feature (ELF) learning framework.
- Elementary-Level Feature Generation: Each image (photo or video keyframe) is first represented as a set of "data points" in an elementary feature space. This can be:
- Concept Scores: Derived from pre-trained models on external datasets (cross-domain learning) or within the target domain. These can be at the image level or region level.
- Low-Level Visual Features: Such as color moments, Gabor texture, and edge histograms, also applied at the image or region level (within-domain learning).
- ELF Construction: A unified procedure, inspired by Bag-of-Features (BoF) techniques, is then used to construct a single feature vector (the ELF) for each event. This involves:
- Calculating pairwise similarities between elementary feature sets using Earth Mover's Distance.
- Constructing a codebook via spectral clustering.
- Representing each event as a codebook-based feature vector.
The paper explores five types of ELFs:
- ELF-KODAK: Within-domain, image-level concept scores from a Kodak consumer dataset.
- ELF-LSCOM: Cross-domain, image-level concept scores from TRECVID videos and the LSCOM ontology.
- ELF-Vis: Within-domain, image-level low-level visual features.
- ELF-RegVis: Within-domain, region-level low-level visual features.
- ELF-RegLHI: Cross-domain, region-level concept scores from the LHI image dataset.
These ELFs are then used to train SVM classifiers. The paper emphasizes the benefits of fusion (early, late, and selective) of these multiple ELFs to achieve better performance.
Datasets
- Kodak Consumer Event Collection: Used for training and testing the final classification models. This dataset contains 1972 events labeled into 10 semantic categories (wedding, birthday, Christmas, parade, picnic, team sport, individual sport, animal, school activity, show).
- Kodak Consumer Video Benchmark Dataset: Used to train the 21 concept detectors for ELF-KODAK.
- TRECVID 2005 Broadcast News Video Set: Used for cross-domain learning to generate ELF-LSCOM. This dataset is associated with a 374-concept LSCOM ontology.
- LHI Image-Parsing Ground-Truth Data Set: Used for cross-domain learning to generate ELF-RegLHI. This dataset contains images with regions labeled for 247 concepts.
Key Insights and Ideas
- The Power of Event-Level Features: Moving beyond image-level analysis to represent an entire event as a single feature vector is crucial for robust classification, especially when dealing with noisy or imperfectly segmented events.
- Complementarity of Features: Combining different types of features (concept scores vs. low-level visual, image-level vs. region-level, cross-domain vs. within-domain) significantly boosts performance. No single feature type is universally best.
- Cross-Domain Learning is Essential: Leveraging large, external, and well-annotated datasets (like TRECVID) is vital for improving classification accuracy, even when the external data distribution differs significantly from the target consumer data. This is especially true for complex events.
- Region-Level Features Add Detail: Incorporating region-level features allows the model to capture finer-grained object information, which complements global image-level features and improves classification.
- Selective Fusion is Key: Simply concatenating all features (early fusion) or averaging classifier outputs (late fusion) can be suboptimal. Selectively combining ELFs based on their performance for specific event categories yields the best results.
Key Results
- Significant Performance Gains: The proposed approach, particularly with selective fusion of multiple ELFs, achieved over 90% MAP gain compared to the previous semantic event classification method (which used only ELF-KODAK).
- Fusion Benefits: Combining different ELFs via fusion methods improved performance by approximately 35% MAP over the best individual ELF. Selective fusion provided a further 70% MAP improvement over the best individual ELF.
- Individual ELF Strengths:
- ELF-RegLHI (region-level, cross-domain concept scores) was the most popular feature, chosen by all semantic event classifiers.
- ELF-LSCOM (image-level, cross-domain concept scores) was effective for events with strong support in TRECVID data (e.g., parade, team sport, wedding), highlighting the value of large ontologies and external data.
- ELF-Vis (image-level visual features) performed well for simple events like "animal."
Limitations and Future Work
- Limitations: The paper acknowledges that while cross-domain learning is powerful, it can be challenging when the data distributions are dramatically different. The "PRED" method used here is flexible but doesn't make strong assumptions about data distribution.
- Future Work: The authors suggest incorporating more types of elementary-level features, exploring additional external data sources and ontologies, and leveraging temporal information from event boundaries. They also propose modeling relationships between visual codebooks for better event description.
Summary updated