Ordering Concepts Based on Common Attribute Intensity

Tatsuya Iwanari, Naoki Yoshinaga, Nobuhiro Kaji, Toshiharu Nishina, Masashi Toyoda, Masaru KitsuregawaIJCAI 2016

Abstract

This paper introduces a novel natural language processing task: the automatic ordering of nominal concepts (e.g., elephant, dog, mouse) according to the intensity of a shared, often subjective, attribute (e.g., large). Unlike objective facts, qualitative attributes like "fashionableness" or "user-friendliness" lack numerical values in standard databases. To address this, the researchers propose a supervised ranking framework that mines "collective wisdom" from a massive corpus of 260 million Japanese blog articles. By integrating diverse linguistic signals—including co-occurrences, dependency relations, similes, and comparative expressions—the system achieves significant alignment with human perception, demonstrating that subjective rankings often carry a strong social consensus.

Introduction: The Challenge of Subjective Ranking

Human decision-making frequently relies on the ability to rank options based on common-sense knowledge. For instance, a person might choose apple juice over lemon juice because of the shared understanding that apples are generally "sweeter." While machines excel at extracting factual data (e.g., "average annual temperature"), quantifying subjective or qualitative attributes remains a significant hurdle.

Existing NLP systems, such as sentiment analysis or fact-based question answering, typically focus on specific entities or objective truths. This research tackles the broader challenge of ranking nominal concepts—general categories rather than specific instances—using attributes that do not have inherent numerical scales. The goal is to capture the "general human perception" reflected in the vast amount of text generated by users online.

Methodology: Integrating Heterogeneous Linguistic Signals

The authors developed a supervised learning approach that synthesizes four distinct types of textual evidence to determine attribute intensity:

  1. Noun-Adjective Co-occurrence: Measuring the frequency with which a concept and an attribute appear together in the same context (e.g., "large" and "elephant").
  2. Noun-Adjective Dependency: Utilizing dependency parsing to identify direct grammatical links between nouns and adjectives (e.g., "The elephant is big").
  3. Similes: Identifying specific patterns like "as [ADJ] as a [NOUN]" (e.g., "brave as a lion"). These expressions often signal extreme intensity and serve as reliable "anchor points" for the ranking.
  4. Comparative Expressions: Directly extracting relative rankings from sentences that compare two concepts (e.g., "Elephants are larger than dogs").

To fuse these signals, two primary frameworks were evaluated:

  • Ranking SVM: A pairwise learning-to-rank method that optimizes the model to correctly order pairs of concepts.
  • Support Vector Regression (SVR): A method that directly estimates the Spearman’s $\rho$ (rank correlation) for a candidate ordering. Due to the factorial explosion of possible permutations, Monte Carlo sampling was employed to make this approach computationally feasible.

Evaluation and Datasets

The study utilized a massive source corpus of 260 million Japanese blog articles, comprising approximately 2 billion sentences crawled since 2006. To evaluate the system, the authors created a benchmark dataset consisting of 35 sets of concepts paired with specific adjectives.

Table 1: Examples of categories, adjectives, and the human-defined gold-standard orderings used for evaluation.

To establish a "gold standard" for these subjective rankings, seven human volunteers ranked the concepts in each set. The final gold standard was defined as the ordering that maximized the average Spearman’s $\rho$ across all participants. Interestingly, the study found a high level of human agreement (average $\rho = 0.75$), suggesting that subjective attributes often have a stable, shared perception within a culture.

Results and Discussion

The proposed Ranking SVM and SVR models significantly outperformed a standard baseline based on Pointwise Mutual Information (PMI).

Table 2: Detailed performance comparison (Spearman’s ρ) between human agreement, the PMI baseline, and the proposed SVM/SVR models across various categories.

Key Findings

  • Ranking SVM Superiority: The Ranking SVM emerged as the most effective method, achieving an average $\rho$ of 0.441, compared to the baseline's 0.274. In specific categories like "MLB team (famous)," the system's performance ($\rho = 0.905$) nearly reached the level of human consensus.
  • The Power of Similes: Although less frequent than simple co-occurrences, similes provided high-quality signals. They effectively identified the extremes of an attribute's intensity scale.
  • Absolute vs. Relative Evidence: The research highlighted that while co-occurrences and dependencies provide evidence of absolute intensity, comparative expressions provide vital relative information. Combining both types of evidence is essential for robust ranking performance.

Conclusion and Future Directions

While the system demonstrated strong performance, several limitations were identified. The model struggled with temporal sensitivity, where the intensity of an attribute (e.g., the "safety" of a country) changes over time, potentially conflicting with a decade's worth of historical blog data. Additionally, data sparseness for niche attributes and the presence of sarcasm or irony in user-generated content remain challenges.

Future work will focus on expanding the system to handle complex phrases (e.g., "easy to cook") and exploring how these conceptual rankings evolve across different demographics and time periods. This research paves the way for more nuanced common-sense reasoning in AI systems.

Summary updated

BibTeX
@inproceedings{iwanari2016ordering,
  author = {Tatsuya Iwanari and Naoki Yoshinaga and Nobuhiro Kaji and Toshiharu Nishina and Masashi Toyoda and Masaru Kitsuregawa},
  title = {{Ordering Concepts Based on Common Attribute Intensity}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure