Lightweight Random Indexing for Polylingual Text Classification (Extended Abstract)

Alejandro Moreo Fernández, Andrea Esuli, Fabrizio SebastianiIJCAI 2018 · DOI

TL;DR

This paper introduces Lightweight Random Indexing (LRI), a novel, efficient method for classifying text documents across multiple languages without relying on machine translation or dictionaries. LRI significantly outperforms existing methods in polylingual text classification while drastically reducing computational cost and memory usage.

What Problem Does This Paper Address?

The internet is awash with information in countless languages. Effectively classifying this multilingual content, known as Polylingual Text Classification (PLC), is a growing challenge. Traditional approaches often struggle with the high dimensionality of multilingual datasets and the lack of shared vocabulary between languages. While some methods use machine translation or external dictionaries to bridge these language gaps, these solutions can be costly, unavailable, or require significant computational resources. This paper aims to develop a PLC method that is both effective and computationally efficient, specifically addressing the need for techniques that are free from external translation or dictionary resources.

Methods

The core of this paper is the introduction of Lightweight Random Indexing (LRI). Random Indexing (RI) is a dimensionality reduction technique that represents words or documents as sparse vectors in a high-dimensional space. It works by assigning a random vector to each word in the original vocabulary. Document vectors are then formed by aggregating the random vectors of the words they contain.

What makes LRI "lightweight" is a specific configuration of the RI parameters:

  • k (non-zero values per random vector): LRI sets k=2. This is the minimum value that allows for distinct encoding of features.
  • n (dimensionality of the projected space): This parameter is explored in the experiments.

This choice of k=2 has several advantages:

  • Sparsity: It maintains a high degree of sparsity in the projected vectors, leading to reduced memory usage and faster computation.
  • Orthogonality: It increases the probability of generating nearly-orthogonal random vectors, which is beneficial for preserving distinct information.
  • Simplicity: Parameter k becomes a fixed constant, eliminating the need for tuning.

The authors compare LRI against several baselines, including:

  • PolyBoW: A multilingual bag-of-words representation.
  • MonoBoW: Naïve monolingual classifiers.
  • RI1%: Standard Random Indexing with 1% non-zero values.
  • Achlioptas mapping: Another random projection method.
  • CL-LSA: Cross-Lingual Latent Semantic Analysis.
  • MDM: Multilingual Domain Models.
  • FS: Feature selection on PolyBoW.

Datasets

The experiments were conducted on two publicly available multilingual corpora:

  • Reuters RCV1/RCV2: This corpus consists of English news stories (RCV1) and news stories in thirteen other languages (RCV2). The stories are comparable at the topic level, meaning they cover similar events but are not direct translations. For their experiments, the authors selected 5 languages (English, Italian, Spanish, French, German) and a subset of classes.
  • JRC-Acquis collections: Mentioned as another dataset used, but detailed experiments are restricted to RCV1/RCV2 in this extended abstract.

Key Insights and Ideas

  • Shared Space for All Languages: Unlike PolyBoW where dimensions are often language-specific, RI (and thus LRI) maps all languages into a common latent space. This means any dimension in the projected space can potentially be informative for all languages simultaneously.
  • "Kernel-Trick" Effect: When the projected space (n) is larger than the number of distinct features for a single language, LRI can exhibit a "kernel-trick" effect. This enlarges the informative space for each language, making them more separable.
  • Sparsity is Key for Efficiency: The choice of k=2 in LRI significantly preserves matrix sparsity. This directly translates to lower memory loads and faster execution times compared to methods that produce dense matrices or less sparse representations.
  • Orthogonality Matters: Higher orthogonality among random vectors, facilitated by LRI's low k, helps in preserving the distinctiveness of features and improving classification performance.

Key Results

  • Improved Accuracy: LRI consistently outperformed all compared algorithms, including PolyBoW and MonoBoW, on the RCV1/RCV2 dataset for both macro- and micro-averaged F1 scores. The improvements were particularly noticeable for less frequent classes (macro-F1).
  • Computational Efficiency: LRI demonstrated significantly reduced computational cost and memory load compared to baselines like RI1%, MDM, and LSA. While LRI's execution time was higher than standard BoW, it was negligible compared to other methods.
  • Scalability: Experiments showed that LRI's performance improved as the dimensionality (n) increased, without a proportional penalty in execution time due to its inherent sparsity.

Limitations and Future Work

  • Limitations: The paper is an extended abstract, so a full discussion of limitations might be in the full paper. However, it's noted that CL-LSA and MDM performed worse than the naïve classifier (MonoBoW) with all features on RCV1/RCV2, which is a surprising result.
  • Future Work: The authors suggest that LRI's effectiveness in preserving sparsity and orthogonality makes it a promising direction for future research in polylingual text classification. The paper also points to the full version of their work ([Moreo Fernández et al., 2016]) for more details and further experiments.

Summary updated

BibTeX
@inproceedings{fernandez2018lightweight,
  author = {Alejandro Moreo Fernández and Andrea Esuli and Fabrizio Sebastiani},
  title = {{Lightweight Random Indexing for Polylingual Text Classification (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/801},
}
Figure