A Document Descriptor using Covariance of Word Vectors

Marwan TorkiACL 2018 · OpenReview

Abstract: Marwan Torki. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

2018.

Introduction

Representing documents as fixed-length vectors is a fundamental task in Natural Language Processing. While word embeddings like Word2Vec or GloVe provide dense representations for individual words, aggregating them into a single document-level descriptor often involves a trade-off between simplicity and expressiveness. Simple averaging (Mean Vectors) is efficient but discards the structural relationships between semantic dimensions. Conversely, distributed representations like Doc2Vec or Skip-Thought capture more complexity but require expensive iterative inference or specialized training.

This paper introduces DoCoV (Document-Covariance), a descriptor that captures second-order statistics of word embeddings to represent text. By calculating the covariance matrix of word vectors within a document, DoCoV provides a "middle ground" that is computationally efficient yet captures the "shape" of the document's semantic distribution.

The DoCoV Method

The core idea behind DoCoV is to represent a document not just by its central tendency (the mean), but by the correlations between different dimensions of its constituent word vectors.

The process involves three main steps:

  1. Observation Matrix: A document with $n$ words and $d$-dimensional embeddings is represented as an $n \times d$ matrix.
  2. Covariance Calculation: A $d \times d$ covariance matrix is computed. This matrix describes the orientation and spread of the word cloud in the embedding space.
  3. Vectorization: Since the covariance matrix is symmetric, the upper triangular part is extracted and flattened into a vector of size $d(d+1)/2$. Off-diagonal elements are scaled by $\sqrt{2}$ to preserve the Frobenius norm in the Euclidean space.
Conceptual visualization of Mean Vectors vs. Covariance (DoCoV) for two different documents.

As illustrated in Figure 1, two documents might have similar mean vectors but distinct covariance "ellipses," reflecting different vocabularies and semantic structures. This "second-order" approach captures how different components of meaning vary together in a specific context.

Experimental Evaluation

The author evaluated DoCoV across several diverse benchmarks, including sentiment analysis (IMDB), semantic textual similarity (SICK, STS-2014), and general text classification (MR, CR, TREC, and Subj).

A major advantage of DoCoV is its zero-inference overhead; unlike Paragraph Vectors, which require gradient descent for new documents, DoCoV is a deterministic, closed-form calculation. Furthermore, the descriptor is highly complementary to traditional baselines.

Key Results

In unsupervised similarity tasks, DoCoV outperformed more complex models like Skip-Thought and Paragraph Vectors in almost every category. In text classification, the synergy between different representation levels proved most effective.

Performance comparison of DoCoV variants against standard document representation models across four classification benchmarks.

As detailed in Figure 2, the combination of DoCoV, Mean Vectors, and Bag-of-Words (COV+Mean+BOW) consistently outperformed complex deep-learning models. On the combined classification benchmark, this hybrid approach achieved an overall accuracy of 86.85%, significantly surpassing standard Doc2Vec (83.8%) and FastSent (78.68%).

Conclusion

DoCoV offers a powerful yet simple alternative for document representation. By leveraging second-order statistics, it captures nuanced semantic relationships that first-order averages miss. While the quadratic growth of the descriptor size ($d^2$) presents a dimensionality challenge for very large embeddings, its performance and computational efficiency make it a strong candidate for large-scale text classification and retrieval tasks.

Summary updated

Abstract

Marwan Torki. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2018.
BibTeX
@inproceedings{torki2018document,
  author = {Marwan Torki},
  title = {{A Document Descriptor using Covariance of Word Vectors}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=Rb_7aqp1dWK},
}
Figure