Using Web Photos for Measuring Video Frame Interestingness

Feng Liu, Yuzhen Niu, Michael GleicherIJCAI 2009 · OpenReview

Abstract: In this paper, we present a method that uses web photos for measuring frame interestingness of a travel video. Web photo collections, such as those on Flickr, tend to contain interesting images because their images are more carefully taken, composed, and selected. Because these photos have already been chosen as subjectively interesting, they serve as evidence that similar images are also interesting. Our idea is to leverage these web photos to measure the interestingness of video frames. Specifically, we measure the interestingness of each video frame according to its similarity to web photos. The similarity is defined based on the scene content and composition. We characterize the scene content using scale invariant local features, specifically SIFT keypoints. We characterize composition by feature distribution. Accordingly, we measure the similarity between a web photo and a video frame based on the co-occurrence of the SIFT features, and the similarity between their spatial distribution. Interestingness of a video frame is measured by considering how many photos it is similar to, and how similar it is to them. Our experiments on measuring frame interestingness of videos from YouTube using photos from Flickr show the initial success of our method.

Introduction

In the world of computer vision, measuring objective qualities like blur, exposure, or face detection has become relatively standard. However, measuring interestingness—that elusive, subjective quality that makes a viewer stop and look—has remained a significant challenge. Traditional video analysis often relies on "low-level" metrics like visual salience or technical quality. Yet, a perfectly sharp photo of a sidewalk isn't necessarily interesting, while a slightly grainy but perfectly composed shot of a landmark is.

In their research, Feng Liu, Yuzhen Niu, and Michael Gleicher propose a novel approach: instead of attempting to teach a computer the abstract principles of beauty, they leverage the "wisdom of the crowd." By looking at what humans have already collectively decided is worth photographing and sharing on platforms like Flickr, they can identify high-quality moments in travel videos that match established aesthetic standards.

Leveraging the Wisdom of the Crowd

The core insight of this work is that web photo collections are not just raw data; they are curated data. People take many photos but typically only upload the ones they find subjectively interesting. This acts as a pre-existing human-labeled dataset for interestingness. If thousands of people have uploaded photos of a specific landmark from a specific angle, that angle is, by definition, interesting.

Overview of the proposed method: leveraging web photo collections to score video frame interestingness.

Methodology

The proposed pipeline follows a "query-and-vote" logic to rank video frames:

  1. Data Gathering: For a given travel video (e.g., of the "Statue of Liberty"), the system queries Flickr for the top 250 photos using the same keyword.
  2. Photo-Frame Registration: The system compares video frames to the web photos using SIFT (Scale Invariant Feature Transform) keypoints. SIFT is robust to changes in scale, rotation, and lighting, making it ideal for matching shaky consumer video to professional still photos.
  3. Geometric Verification: To ensure matches are valid, the system uses the Fundamental Matrix and RANSAC to verify that the spatial relationship between points in the video frame matches the photo.
  4. Temporal Coherence: To maintain computational efficiency, the system builds "SIFT feature tracks" across consecutive video frames, allowing it to propagate interestingness scores through time.

The Interestingness Metric

A frame’s score is determined by two primary criteria:

  • Content (C1): How many features does the frame share with the web photo set?
  • Composition (C2): How similar is the spatial distribution of those features? This implicitly captures artistic conventions like framing and perspective by checking if the landmark is positioned similarly to popular web photos.

Experimental Results

The researchers evaluated their method using 30 travel videos from YouTube and approximately 2,400 photos from Flickr across 10 different landmarks, including the Eiffel Tower, the Golden Gate Bridge, and the Taj Mahal.

To validate the subjective metric, a user study was performed with five participants. They rated how well the algorithm's "interestingness curve" matched their own judgment on a scale of 1 to 5. The average score was a high 4.5/5, indicating strong alignment with human perception.

Frame interestingness curves for three landmark videos, showing the relationship between web photo similarity (WPV) and visual quality (QLT).

As shown in the results above, the system successfully identifies iconic views and well-composed shots. Interestingly, the study found that "interestingness" does not correlate strongly with traditional visual quality metrics (like blur or blockiness). A frame can be technically imperfect but still highly interesting because of its content and composition.

Limitations and Future Work

While successful, the method faces a few hurdles. Significant motion blur can prevent SIFT features from being detected, causing the system to miss interesting frames during fast camera pans. Additionally, the system may occasionally prioritize "context" (like a lush forest) over the "subject" (a plain monument) if the context has more detectable features.

Currently, the method relies on well-known landmarks with abundant web photos. Future work could explore incorporating factors like the presence of people or specific motifs to broaden the definition of interestingness beyond famous landmarks to everyday videos.

Summary updated

Abstract

In this paper, we present a method that uses web photos for measuring frame interestingness of a travel video. Web photo collections, such as those on Flickr, tend to contain interesting images because their images are more carefully taken, composed, and selected. Because these photos have already been chosen as subjectively interesting, they serve as evidence that similar images are also interesting. Our idea is to leverage these web photos to measure the interestingness of video frames. Specifically, we measure the interestingness of each video frame according to its similarity to web photos. The similarity is defined based on the scene content and composition. We characterize the scene content using scale invariant local features, specifically SIFT keypoints. We characterize composition by feature distribution. Accordingly, we measure the similarity between a web photo and a video frame based on the co-occurrence of the SIFT features, and the similarity between their spatial distribution. Interestingness of a video frame is measured by considering how many photos it is similar to, and how similar it is to them. Our experiments on measuring frame interestingness of videos from YouTube using photos from Flickr show the initial success of our method.
BibTeX
@inproceedings{liu2009using,
  author = {Feng Liu and Yuzhen Niu and Michael Gleicher},
  title = {{Using Web Photos for Measuring Video Frame Interestingness}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2009},
  url = {https://openreview.net/forum?id=SJWP9Bz_WB},
}
Figure