OTTER: Open-Tagging via Text-Image Representation for Multi-Modal Understanding

Jieer Ouyang, Xiaoneng Xiang, Zheng Wang, Yuting Ding2025 · arXiv

Introduction

In our digital lives, we generate a staggering amount of heterogeneous media—photos, documents, and screenshots. Organizing this content effectively requires a tagging system that is both consistent (for broad categories like "Life Moments") and flexible (for specific user needs like "Wedding Planning").

Existing solutions typically fall into two categories: closed-set systems, which are stable but rigid, and open-vocabulary systems (like CLIP), which handle any tag but often struggle with semantic precision and grounding textual labels in visual evidence. OTTER (Open-Tagging via Text-Image Representation) bridges this gap by harmonizing a stable taxonomy with evolving user-defined labels.

The OTTER Framework

OTTER introduces a two-tier tagging framework supported by a specialized model architecture designed for multi-modal fusion.

Model Architecture and Fusion

The model utilizes a multi-head attention mechanism where label embeddings, processed by a shared text encoder, act as queries. These queries attend over fused features from the input data. For visual inputs, OTTER employs a vision backbone (such as ViT) but adds a critical component: Optical Character Recognition (OCR). By extracting text from within images—such as headlines in a screenshot or labels on a product—the model significantly boosts tagging accuracy.

The OTTER model architecture integrates image and text encoders with OCR-derived keywords through a multi-head attention mechanism.

For text-only or hybrid inputs, the model processes titles and body content. These features are combined using additive fusion, which the authors found more effective than simple concatenation for preserving complementary information and reducing overfitting to redundant high-dimensional data.

Hybrid Training Strategy

The system is trained on a two-tier taxonomy: six high-level "Predefined Tags" (e.g., Career & Business, Tech Frontiers) and a dynamic set of "User-defined Labels." During training, the model is exposed to both fixed labels and sampled open-set labels, helping it learn robust decision boundaries between true positive tags and negative candidates.

Datasets and Evaluation

The research utilized two primary datasets for evaluation:

  1. Otter Dataset: A large-scale, proprietary dataset organized hierarchically and annotated using a hybrid pipeline of LLM-based initial tagging (ChatGPT) followed by human refinement.
  2. Favorite Dataset: A collection of image-text pairs from real-world user scenarios, manually labeled to provide a high-quality ground truth for daily-life content.

Key Results

OTTER demonstrates a significant performance lead over state-of-the-art competitive baselines, including CLIP, SIGLIP, and Qwen2VL.

Performance comparison of OTTER against baseline models across Predefined and Open Tag categories.

As shown in the results, OTTER achieved an F1 score of 0.81 on the Otter dataset, surpassing CLIP by a margin of 0.10. Most notably, the model's performance on user-defined, open-set labels was nearly perfect, reaching an F1 of 0.99 on the Otter dataset and 0.97 on the Favorite dataset. Unlike some baselines that sacrificed recall for precision, OTTER maintained a balanced profile, making it highly reliable for real-world retrieval tasks.

Conclusion

OTTER successfully addresses the limitations of rigid classification systems by treating labels as queries in an attention mechanism. This decouples the label space from the feature space, allowing the model to "look" for specific concepts regardless of whether they were in the original training set. While challenges remain in managing complex label co-occurrences and scaling to massive candidate sets, OTTER establishes a powerful new baseline for open-set multi-modal understanding in personal information management systems.

Summary updated

Abstract

We introduce OTTER, a unified open-set multi-label tagging framework that harmonizes the stability of a curated, predefined category set with the adaptability of user-driven open tags. OTTER is built upon a large-scale, hierarchically organized multi-modal dataset, collected from diverse online repositories and annotated through a hybrid pipeline combining automated vision-language labeling with human refinement. By leveraging a multi-head attention architecture, OTTER jointly aligns visual and textual representations with both fixed and open-set label embeddings, enabling dynamic and semantically consistent tagging. OTTER consistently outperforms competitive baselines on two benchmark datasets: it achieves an overall F1 score of 0.81 on Otter and 0.75 on Favorite, surpassing the nextbest results by margins of 0.10 and 0.02, respectively. OTTER attains near-perfect performance on open-set labels, with F1 of 0.99 on Otter and 0.97 on Favorite, while maintaining competitive accuracy on predefined labels. These results demonstrate OTTER's effectiveness in bridging closed-set consistency with open-vocabulary flexibility for multi-modal tagging applications.
BibTeX
@misc{ouyang2025otter,
  author = {Jieer Ouyang and Xiaoneng Xiang and Zheng Wang and Yuting Ding},
  title = {{OTTER: Open-Tagging via Text-Image Representation for Multi-Modal Understanding}},
  year = {2025},
  eprint = {2510.00652},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.00652},
}
Figure