Abstract
A significant portion of the world's text is tagged by readers on social bookmarking websites. Credit attribution is an inherent problem in these corpora because most pages have multiple tags, but the tags do not always apply with equal specificity across the whole document. Solving the credit attribution problem requires associating each word in a document with the most appropriate tags and vice versa. This paper introduces Labeled LDA, a topic model that constrains Latent Dirichlet Allocation by defining a one-to-one correspondence between LDA's latent topics and user tags. This allows Labeled LDA to directly learn word-tag correspondences. We demonstrate Labeled LDA's improved expressiveness over traditional LDA with visualizations of a corpus of tagged web pages from del.icio.us. Labeled LDA outperforms SVMs by more than 3 to 1 when extracting tag-specific document snippets. As a multi-label text classifier, our model is competitive with a discriminative baseline on a variety of datasets.
Introduction: The Credit Attribution Problem
In modern digital ecosystems, documents are rarely characterized by a single theme. A single article on a social bookmarking site like del.icio.us might be tagged with "politics," "transportation," and "finance." While these labels provide a high-level summary, they do not indicate which specific words or sentences within the text triggered each tag. This is known as the credit attribution problem.
Traditional Latent Dirichlet Allocation (LDA) is an effective unsupervised tool for discovering latent topics, but these topics often lack interpretability and do not necessarily align with human-provided tags. Labeled LDA (L-LDA) addresses this by incorporating supervision directly into the generative process, creating a direct link between specific labels and specific sets of words.
The Labeled LDA Model
Labeled LDA is a generative model that constrains the flexibility of standard LDA to match the supervision of user-provided tags. The core innovation lies in a one-to-one correspondence between topics and tags:
- Topic-Tag Mapping: The number of topics is set to exactly equal the number of unique tags in the corpus.
- Constrained Mixture: During document generation, the model restricts the document's topic distribution ($\theta$) to only those topics that correspond to the document's observed labels.
- Projection Matrix: A document-specific projection matrix transforms a global Dirichlet prior into a lower-dimensional prior containing only the relevant tags.
By forcing a document to be a mixture of only its own tags, L-LDA directly learns word-tag correspondences using standard inference techniques like collapsed Gibbs sampling.
Key Insights: Interpretability and Disambiguation
Because each topic is explicitly tied to a tag, the resulting word distributions are immediately interpretable. Unlike standard LDA, where a researcher must "guess" what a topic represents, an L-LDA topic for "Physics" is inherently the "Physics" tag.
Furthermore, L-LDA excels at contextual disambiguation. In a document tagged with both "Design" and "Programming," the model can correctly attribute the word "designed" to the "Design" tag and technical terms like "CMS" to the "Programming" tag based on the overall topic mixture. This allows for the creation of "heat maps" that highlight which sentences are most relevant to specific tags.
Experimental Results
The researchers evaluated L-LDA on the del.icio.us tag dataset (4,000 pages, ~90% multi-labeled) and the Yahoo Directory.
Snippet Extraction
One of the most compelling applications of L-LDA is tag-specific summarization. By identifying the most "tag-dense" regions of a document, the model can extract snippets tailored to a user's specific interest. In human evaluation trials, L-LDA's snippets were judged superior to those generated by Support Vector Machines (SVMs) by a ratio of more than 3 to 1.
Multi-label Classification
Despite being a generative model, L-LDA proved highly competitive with discriminative one-vs-rest SVMs. On the del.icio.us dataset, L-LDA significantly outperformed SVMs in Micro-F1 scores, achieving 52.12% compared to 39.33% for the SVM baseline.
Conclusion
Labeled LDA provides a robust framework for solving the credit attribution problem in multi-labeled corpora. By constraining the latent topic space to match observed labels, it achieves high interpretability and state-of-the-art performance in tag-specific summarization. Future work may involve incorporating label correlations (similar to Correlated Topic Models) and improving inference efficiency for completely unlabeled data.
Summary updated