Abstract
Automatic image annotation often suffers from "semantic redundancy," where models output synonymous tags (e.g., "person" and "people") that fail to capture the full breadth of a scene. This paper presents $D^2IA$ (Diverse and Distinct Image Annotation), a novel framework that leverages Generative Adversarial Networks (GANs) and Determinantal Point Processes (DPP) to produce tag sets that are both semantically unique and representative of multiple human perspectives. By incorporating semantic hierarchies, $D^2IA$ mimics the collective descriptive power of human annotators, significantly improving the informativeness and diversity of generated tags.
Introduction
Standard image tagging algorithms primarily optimize for relevancy—how accurately a tag describes an image. However, this narrow focus leads to two significant drawbacks: semantic redundancy and a lack of diversity. Redundancy occurs when a model outputs multiple tags with overlapping meanings, such as "building" and "house," providing little additional information. Lack of diversity refers to the model's tendency to provide a single "best" set of tags, ignoring the varied perspectives different human observers might have.
The authors propose the $D^2IA-GAN$ to address these issues. They define "distinctness" as the internal semantic uniqueness of a single tag set and "diversity" as the ability of the model to generate different, relevant perspectives across multiple samples. This approach simulates how an ensemble of human annotators might describe the same image.
Methods
The $D^2IA-GAN$ architecture combines generative modeling with point process theory to balance relevancy, distinctness, and diversity.
Determinantal Point Processes (DPP)
To ensure distinctness, the model employs a DPP. DPPs are probabilistic models that favor subsets of items with high diversity by assigning higher probabilities to sets of items that are "repulsive" or semantically distant. The authors integrate a semantic hierarchy (weighted semantic paths) into the DPP to prevent the selection of multiple tags from the same branch of a knowledge tree.
Generative Adversarial Network (GAN)
To achieve diversity, the framework uses a GAN. The Generator takes image features and a random noise vector to produce a tag subset, where the noise allows for different relevant perspectives in each run. The Discriminator is trained to distinguish between these generated sets and actual ground-truth human annotations.
Policy Gradient Training
Since the sampling process in a DPP is discrete and non-differentiable, the authors utilize Policy Gradient (PG) reinforcement learning. This allows the generator to be optimized based on rewards derived from the discriminator's feedback and image-tag relevancy scores.
Experiments and Results
The model was evaluated on the ESP Game and IAPRTC-12 datasets, both of which include semantic hierarchy information necessary for the weighted semantic paths.
- Quantitative Performance: On the ESP Game dataset (using 3 tags), $D^2IA$ achieved a semantic F1-score ($F_{1-sp}$) of 34.93%, surpassing the state-of-the-art DIA method (33.61%). The improvement was even more significant in "ensemble" scenarios, where $D^2IA$ showed a 26.57% relative improvement in semantic recall over DIA.
- Human Evaluation: In blind tests, human participants preferred $D^2IA$ tag sets over those from competing models in over 60% of cases across both datasets, citing better comprehensiveness and less redundancy.
- Consistency: A high correlation was observed between automated semantic metrics and human judgment, validating the proposed evaluation framework as a reliable proxy for human preference.
Conclusion
$D^2IA-GAN$ successfully bridges the gap between machine-generated tags and human-like descriptions by prioritizing semantic distinctness and diversity. While the model currently relies on pre-defined semantic hierarchies, its success in simulating the "ensemble" of human perspectives marks a significant step forward in image annotation. Future work may explore extending this approach to more complex tasks like natural language captioning and video description, potentially reducing the dependency on fixed hierarchies.
Summary updated