When humans describe a scene, they naturally filter out mundane background noise to focus on salient elements. For instance, a photo of a family dinner is typically described as "a family eating together in a kitchen," rather than a list of every visible object like "a fork, a salt shaker, a chair leg, and a shoe in the corner." This process of content selection is a fundamental component of natural language generation that remains a significant challenge for automated systems.
In their paper, "Don’t Mention the Shoe!", Josiah Wang and Robert Gaizauskas from the University of Sheffield address this challenge by framing content selection as a Learning to Rank (LTR) problem. By ranking detected objects based on their likelihood of being mentioned by humans, their approach achieves state-of-the-art performance, nearing human-level selection accuracy.
The Content Selection Problem
Most automatic image description systems operate in an "end-to-end" fashion, making it difficult to diagnose whether poor captions result from computer vision failures (not seeing objects) or selection failures (not knowing which objects are important).
This research isolates the content selection sub-problem. Given an image where objects have already been perfectly localized using "gold standard" bounding boxes, the system must decide which specific boxes should be included in the final description. The challenge lies in the fact that "importance" is subjective and highly contextual; a fork is vital in a close-up of a meal but irrelevant in a wide shot of a crowded restaurant.
Methodology: Ranking and Stopping
The authors propose a two-stage pipeline consisting of Ranking and Stopping.
1. Learning to Rank
Every object in an image is treated as a candidate and ordered by "mention-ability" using three LTR approaches:
- Pointwise (Random Forests): Treats each object independently.
- Pairwise (Ranking SVM): Optimizes the relative order of pairs of objects.
- Listwise (Coordinate Ascent): Optimizes the entire list based on evaluation metrics like NDCG.
2. Feature Engineering
To rank these objects, the authors extracted three types of data:
- Geometric: Box size and distance from the image center.
- Semantic: One-hot category labels and Word2Vec-based "synset embeddings" to capture the meaning of the object.
- Visual: Deep features (FC7 layer from VGG-16) representing the actual pixels within the box.
3. Stopping Criteria
Once a ranked list is generated, the system must decide where to draw the line. The authors tested fixed cut-offs (e.g., top $k$ items), absolute score thresholds, and a "relative" strategy that identifies the largest statistical gap between items in the ranked list.
Key Insights
- Semantics Trump Pixels: The most powerful predictors were concept labels rather than visual textures. Knowing an object is a "person" or a "dog" is a much stronger signal for mention-ability than the raw visual data of that object.
- The Pointwise Surprise: Surprisingly, the simpler Random Forest (pointwise) approach outperformed more complex pairwise and listwise methods. This suggests that an object's inherent importance (e.g., "people are usually important") often outweighs complex contextual relationships in this dataset.
- Size and Saliency: Larger objects and those closer to the center are significantly more likely to be mentioned, confirming long-standing theories in visual saliency.
Results
The study utilized the ImageCLEF 2015 Scalable Image Annotation dataset, comprising 950 images with gold-standard bounding boxes mapped to WordNet synsets.
As shown in the results table, the proposed system significantly advanced the benchmark. The best model (Random Forest with an absolute threshold) achieved an F-score of 0.70, surpassing the previous state-of-the-art (0.56) and the winning entry of the original 2015 challenge (0.53). Most notably, the system's performance is remarkably close to the human upper-bound of 0.74, suggesting the LTR framework effectively captures the logic humans use when describing scenes.
Conclusion and Future Work
While the system achieves near-human performance, it currently assumes "clean" input with perfect bounding boxes. In real-world applications, these would be provided by noisy object detectors. Future research will focus on integrating this ranking system with automated detectors and exploring stronger textual features to capture the nuances of category prioritization. This LTR framework holds promise for various Natural Language Generation tasks where content selection remains a primary bottleneck.
Summary updated