Introduction
In our digital lives, we generate a staggering amount of heterogeneous media—photos, documents, and screenshots. Organizing this content effectively requires a tagging system that is both consistent (for broad categories like "Life Moments") and flexible (for specific user needs like "Wedding Planning").
Existing solutions typically fall into two categories: closed-set systems, which are stable but rigid, and open-vocabulary systems (like CLIP), which handle any tag but often struggle with semantic precision and grounding textual labels in visual evidence. OTTER (Open-Tagging via Text-Image Representation) bridges this gap by harmonizing a stable taxonomy with evolving user-defined labels.
The OTTER Framework
OTTER introduces a two-tier tagging framework supported by a specialized model architecture designed for multi-modal fusion.
Model Architecture and Fusion
The model utilizes a multi-head attention mechanism where label embeddings, processed by a shared text encoder, act as queries. These queries attend over fused features from the input data. For visual inputs, OTTER employs a vision backbone (such as ViT) but adds a critical component: Optical Character Recognition (OCR). By extracting text from within images—such as headlines in a screenshot or labels on a product—the model significantly boosts tagging accuracy.
For text-only or hybrid inputs, the model processes titles and body content. These features are combined using additive fusion, which the authors found more effective than simple concatenation for preserving complementary information and reducing overfitting to redundant high-dimensional data.
Hybrid Training Strategy
The system is trained on a two-tier taxonomy: six high-level "Predefined Tags" (e.g., Career & Business, Tech Frontiers) and a dynamic set of "User-defined Labels." During training, the model is exposed to both fixed labels and sampled open-set labels, helping it learn robust decision boundaries between true positive tags and negative candidates.
Datasets and Evaluation
The research utilized two primary datasets for evaluation:
- Otter Dataset: A large-scale, proprietary dataset organized hierarchically and annotated using a hybrid pipeline of LLM-based initial tagging (ChatGPT) followed by human refinement.
- Favorite Dataset: A collection of image-text pairs from real-world user scenarios, manually labeled to provide a high-quality ground truth for daily-life content.
Key Results
OTTER demonstrates a significant performance lead over state-of-the-art competitive baselines, including CLIP, SIGLIP, and Qwen2VL.
As shown in the results, OTTER achieved an F1 score of 0.81 on the Otter dataset, surpassing CLIP by a margin of 0.10. Most notably, the model's performance on user-defined, open-set labels was nearly perfect, reaching an F1 of 0.99 on the Otter dataset and 0.97 on the Favorite dataset. Unlike some baselines that sacrificed recall for precision, OTTER maintained a balanced profile, making it highly reliable for real-world retrieval tasks.
Conclusion
OTTER successfully addresses the limitations of rigid classification systems by treating labels as queries in an attention mechanism. This decouples the label space from the feature space, allowing the model to "look" for specific concepts regardless of whether they were in the original training set. While challenges remain in managing complex label co-occurrences and scaling to massive candidate sets, OTTER establishes a powerful new baseline for open-set multi-modal understanding in personal information management systems.
Summary updated