TL;DR
This paper tackles the challenge of automatically extracting "composite properties" (like "long ears" for rabbits) from text, which are crucial for human-like concept descriptions but often overlooked by existing methods. They propose and evaluate several corpus-based techniques to identify these modifiers, showing that combining general and concept-specific co-occurrence information yields the best results.
What Problem Does This Paper Address?
In cognitive science, understanding how humans represent concepts often involves studying how they describe them. These descriptions frequently include "properties," such as "rabbits have fur" or "birds have wings." While many computational approaches can extract these simple property-concept pairs from text, they struggle with "composite properties." These are properties where a modifier is applied to a part, like "long ears" instead of just "ears." These composite properties are common in human descriptions and are important for capturing nuanced concept representations. Existing methods often strip modifiers to simplify the task, losing valuable information. This paper aims to bridge this gap by developing methods to automatically identify these cognitively salient modifiers for concept parts.
Methods
The authors propose and evaluate several methods for extracting adjective modifiers for concept parts from large text corpora. Their core idea is to leverage co-occurrence statistics, but with a twist: they consider both general co-occurrences of a modifier and a part (contextless) and co-occurrences within a specific concept's context (in-context).
Here's a breakdown of their approaches:
- Frequency-based methods:
- Contextless Frequency: Ranks modifiers based on their raw co-occurrence frequency with a part noun in the entire corpus.
- Log-Likelihood: Similar to raw frequency but uses log-likelihood scores to better account for modifiers that are rare but highly specific to a part.
- In-Context Frequency: Ranks modifiers based on their co-occurrence with a part noun only when the target concept is also present in a surrounding window of text. This aims to capture concept-specific modifiers.
- Combination Methods:
- Summed Log-Rescaled Frequencies: Combines the contextless and in-context lists by summing their log-rescaled frequencies.
- Productwise Combination of Frequencies: Combines the lists by multiplying their frequencies (with a back-off for missing in-context data). This method aims for an "intersective" effect, boosting modifiers that are both generally plausible and concept-specifically relevant.
- Cosine-Based Re-Ranking: This method takes the productwise combined list and re-ranks modifiers based on their similarity to other modifiers. It uses random indexing to create vector representations of modifiers and calculates cosine distances to identify modifiers that co-occur with similar sets of nouns, aiming to group modifiers by attribute type.
Datasets
- German and Italian Feature Norms: These are collections of subject-generated concept descriptions, serving as the "gold standard" for evaluating their methods. They contain concept-modifier-part triples (e.g., "onion: brown peel").
- German and Italian WaCky Web Corpus: Large web corpora (around 1.2 billion tokens each) used for extracting co-occurrence statistics.
- Plausibility Judgement Dataset: A new benchmark created by the authors where human participants rated the plausibility of generated concept-modifier-part triples. This dataset was used to evaluate how well their extracted modifiers align with human perception, not just production.
Key Insights and Ideas
- Composite properties are key: The paper strongly emphasizes that composite properties (like "long ears") are a significant and often overlooked aspect of human concept descriptions, making up about a third of part descriptions in their feature norms.
- Context matters, but not exclusively: Simply looking at co-occurrences within a concept's context (in-context) leads to low recall due to data sparsity. Conversely, only considering general co-occurrences (contextless) misses concept-specific nuances.
- Intersection is powerful: The "productwise combination" method, which effectively intersects general and concept-specific co-occurrence information, performs significantly better than methods relying on only one source or simple additive combinations. This suggests that modifiers need to be both generally plausible for a part and specifically relevant to the concept.
- Production vs. Perception: There's a surprisingly small overlap between what people produce in concept descriptions and what they judge as plausible modifiers. This highlights that algorithms might uncover useful modifiers that humans don't spontaneously produce, and vice-versa.
- Re-ranking complexity doesn't guarantee improvement: The more complex cosine-based re-ranking method, intended to group modifiers by attribute type, did not significantly outperform the simpler productwise combination method in their evaluations.
Key Results
- Best performing method: The "productwise combination" of contextless and in-context frequency lists achieved the best performance in terms of recall and precision on both German and Italian norms.
- Performance metrics: On German norms, the best models achieved around 20% precision at 30% recall, significantly outperforming a random baseline (below 1% precision).
- Cross-lingual generalization: The methods showed similar performance on Italian data, indicating good cross-lingual generalization capabilities.
- Plausibility judgements: When evaluated on human plausibility judgements, the productwise combination method also performed well, with modifiers accepted by at least 75% of raters considered plausible.
- Overlap analysis: The study found a notable difference between produced and perceived modifiers, with 46 triples being produced but not accepted, and 42 being accepted but not produced.
Limitations and Future Work
- Modifier types: The current work focuses solely on adjective modifiers. Future work could extend to numerals (e.g., "four wheels") and explore whether a part itself is sufficient to describe a concept (e.g., "big ears" vs. "trunk").
- Evaluation granularity: The authors suggest that a more exact performance measure might involve requiring a majority of participants to produce a modifier for the gold standard, or having participants judge produced modifiers to filter out unlikely ones.
- Re-ranking effectiveness: The cosine-based re-ranking method, while conceptually appealing, did not show a significant advantage. Further research could explore alternative re-ranking strategies or machine learning classifiers.
- Extracting other relation types: The ultimate goal is to extract not only parts but also other types of relations (category, behavior, function, etc.), which this paper lays the groundwork for.
- Specialized corpora: Crawling the web for concept-specific text passages to build specialized corpora is suggested as a potential avenue for richer information.
Summary updated