TL;DR
This paper tackles the challenge of subjectivity detection in "resource-lean" languages like Persian, by leveraging readily available English resources. They propose and evaluate two cross-lingual approaches: a language-model-based method and a learning-to-rank method, demonstrating that the latter significantly outperforms the former, especially when using individual terms as queries.
What Problem Does This Paper Address?
The explosion of user-generated content on the web, particularly on platforms like Web 2.0, has led to a surge in textual data expressing opinions and sentiments. While significant research has focused on analyzing this "subjective" data in resource-rich languages like English, a substantial gap exists for languages with fewer available linguistic resources (e.g., sentiment lexicons, annotated corpora). This paper aims to bridge this gap by investigating how to effectively use English resources to perform subjectivity detection on a resource-lean language, specifically Persian. The core challenge is overcoming the inherent difficulties of cross-lingual transfer, including translation ambiguities and the lack of direct parallel data.
Methods
The authors propose and compare two primary cross-lingual subjectivity detection methods:
- Language-Model Based Method: This approach, adapted from prior work, builds statistical language models for subjective and objective text from English training data. For a Persian test document, its language model is compared against these English models using KL-divergence to compute a subjectivity score. Crucially, this method explores two translation directions:
- English to Persian (Source Translation): English reference language models are translated into Persian.
-
Persian to English (Target Translation): Persian test documents (or their unigrams) are translated into English to be compared with the English reference models. The method also considers both word-level and document-level translation units.
-
Learning-to-Rank (LTR) Method: This approach frames subjectivity detection as a ranking problem. It utilizes a sentiment lexicon (a list of subjective terms) as a query. The goal is to rank test documents based on their relevance to these subjective queries. The paper explores two ways of using the query list:
- List Query Level: The entire list of subjective terms is treated as a single query.
- Term Query Level: Each subjective term from the lexicon is treated as an individual query. The LTR method employs various features, including term frequency, inverse document frequency, BM25, and KL-divergence, all adapted for the cross-lingual setting through probabilistic translations. The authors use SVM-rank and RankLib for implementation.
Datasets
- English Dataset: A dataset of 5000 movie reviews (subjective) from Rotten Tomatoes and 5000 movie summaries (objective) from the Internet Movie Database. The average document length is 11 words.
- Persian Dataset: A newly constructed dataset for this paper. Subjective documents were gathered from movie critic websites (e.g., naghdefarsi.com, myturn.blogsky.com), totaling over 7500 documents. Objective documents were sourced from Wikipedia, focusing on factual information like "Actors/Actresses," "Awards," and "Movie Story" sections, totaling 3500 documents. The average document length is 83 words.
Key Insights and Ideas
- Learning-to-Rank Outperforms Language Models: The LTR approach consistently achieves higher performance (measured by MAP) than the language-model-based method, especially when using individual terms as queries. This suggests that framing subjectivity detection as a ranking task with rich features is more effective for cross-lingual transfer.
- Term-Level Queries are Superior: Treating each subjective term as a separate query in the LTR method yields significantly better results than using the entire list of subjective terms as a single query. This is attributed to mitigating the impact of translation errors; an error in one term's translation only affects that specific query, not the entire document's score.
- Translation Direction Matters: The direction of translation (English-to-Persian vs. Persian-to-English) impacts performance, with some configurations showing better results when translating from Persian to English (e.g., word-level LM) and others from English to Persian (e.g., term-level LTR).
- Translation Unit Impact: Both word-level and document-level translations are explored. Word-level translation, particularly in the LTR approach, seems to be more granular and effective. Document-level translation can be simpler but might lose nuance.
- Language Differences Pose Challenges: The paper acknowledges significant structural and grammatical differences between Persian and English (e.g., verb placement, adjective usage) that complicate cross-lingual transfer.
Key Results
- Language-Model Based Method:
- Word-level translation from Persian to English (LM-PTOE-W) achieved a MAP of 0.849, outperforming translation from English to Persian (LM-ETOP-W, MAP 0.693).
- Document-level translation from English to Persian (LM-ETOP-D, MAP 0.800) performed better than Persian to English (LM-PTOE-D, MAP 0.483).
- Learning-to-Rank Method:
- Using individual terms as queries (AD-ETOP-TQ, MAP 0.929) significantly outperformed using the entire list as one query (AD-ETOP-LQ, MAP 0.675) when translating from English to Persian.
- The best overall result was achieved by the AD-ETOP-TQ run with a MAP of 0.929.
- When comparing LTR algorithms (Random Forests, ADARank, Coordinate Ascent) with Persian as the training set and English as the test set, Coordinate Ascent achieved the highest MAP (0.860).
- Overall Comparison: The paper concludes that the learning-to-rank based method, particularly with term-level queries and appropriate translation direction, outperforms the language-model based approach.
Limitations and Future Work
- Translation Tool Dependency: The performance of both methods is heavily reliant on the quality of the machine translation tools used (Google Translate for LM, Moses for LTR).
- Dataset Specificity: The constructed Persian dataset is specific to the movie domain. The generalizability of the findings to other domains or other resource-lean languages is not extensively explored.
- Future Work: The authors suggest studying the impact of translation on subjectivity detection in other resource-lean languages as a future direction. They also mention exploring other algorithms for the learning-to-rank approach.
Summary updated