TL;DR
This paper tackles the challenge of learning interpretable, "conceptual spaces" from data, where dimensions correspond to meaningful semantic features. They propose a hierarchical approach to discover these features, arguing that single-space embeddings are suboptimal because semantic features are often only relevant to subsets of entities. Their method effectively learns domain-specific quality dimensions, leading to more meaningful representations.
What Problem Does This Paper Address?
Traditional methods for learning entity embeddings (like word embeddings) produce vector representations where dimensions lack clear semantic meaning. This contrasts with "conceptual spaces" from cognitive science, where dimensions (called "quality dimensions") represent salient semantic features (e.g., hue, saturation for color). While previous work has shown that semantic features can sometimes be captured as directions in learned embeddings, a key limitation is that these features are often only relevant to a subset of entities. For example, a feature about "adherence to a book" is meaningful for movie adaptations but not for all movies. Learning these features in a single, global vector space can lead to suboptimal representations because the dimensions try to accommodate features that aren't universally applicable. The paper aims to address this by learning features in a hierarchical manner, allowing for domain-specific quality dimensions.
Methods
The core of the paper builds upon an existing unsupervised method ([Derrac and Schockaert, 2015]) for identifying semantic features as directions in vector spaces. This method uses linear classifiers to find words that can separate entities based on whether they appear in the entity's description. The key innovation here is a hierarchical approach.
- Primary Feature Learning: They first apply the existing method to the entire dataset to discover "primary features." These features act as top-level discriminators, effectively splitting the data into domains. Instead of using k-means for clustering feature directions (as in the original work), they use affinity propagation, which doesn't require pre-specifying the number of clusters – crucial for the hierarchical setting.
- Sub-Feature Learning: For each primary feature, they then re-apply the feature discovery method, but only on the subset of entities that exhibit that primary feature. This allows them to discover "sub-features" that are specific to that domain.
- Orthogonal Decomposition (Optional Variant): They also propose an optional variant where sub-feature directions are made orthogonal to their parent primary feature direction. This aims to ensure that sub-features capture complementary information to the primary feature, reducing redundancy.
Datasets
The authors evaluate their method on five datasets:
- Movies: Text descriptions from movie reviews.
- Place types: Text descriptions from bags of Flickr tags.
- Organisations: Text descriptions from Wikipedia articles, predicting Wikidata attributes.
- Buildings: Text descriptions from Wikipedia articles, predicting Wikidata attributes.
- Music bands: A new dataset focusing on music bands.
These datasets are domain-specific, as meaningful features are context-dependent. For entity embeddings, they used 100-dimensional vectors obtained via Multi-Dimensional Scaling (MDS) and Doc2Vec.
Key Insights and Ideas
- Semantic features are often hierarchical: The paper provides strong evidence that the linear regularities in learned embeddings exhibit an inherent hierarchical structure, mirroring how humans conceptualize the world.
- Single-space embeddings are suboptimal for domain-specific features: Trying to learn features that are only relevant to a subset of entities in a global vector space leads to poorer representations. Hierarchical learning allows features to be specialized.
- Affinity propagation is beneficial for unsupervised clustering: Using affinity propagation over k-means for clustering feature directions proved more effective, especially in a hierarchical setting where the number of sub-clusters is unknown.
- Sub-features can capture nuanced distinctions: Even when primary features are well-defined, sub-features can differentiate finer-grained categories within that domain (e.g., different types of "gore" movies).
- Learned representations can bridge symbolic and sub-symbolic: The discovered features, while data-driven and potentially "messier" than idealized conceptual spaces, can still be used to build interpretable models (like decision trees) and bridge the gap between vector representations and symbolic knowledge.
Key Results
- Sub-feature learning consistently outperforms primary-only features: Across most datasets and evaluation metrics (F1 score for depth-1 and depth-3 decision trees), their proposed "Sub" method (hierarchical learning) significantly outperforms the "Primary" method (learning features only at the top level).
- Competitive performance against baselines: The "Sub" method generally outperforms random feature selection and Agglomerative Hierarchical Clustering (AHC) baselines.
- Orthogonal variant shows promise: The "Ortho" variant (using orthogonal sub-feature directions) performed very well, even outperforming "Sub" on the "place types" dataset, suggesting that enforcing complementary information can be beneficial, especially in noisy domains.
- Robustness to feature count: The improvement from sub-features is robust even when varying the total number of features used.
- Qualitative analysis supports interpretability: Examples of learned primary and sub-features for movies demonstrate their semantic meaningfulness and ability to capture nuanced distinctions, enabling the construction of interpretable decision trees.
Limitations and Future Work
- Two-level hierarchy sufficiency: The paper primarily focuses on two-level hierarchies, stating they were sufficient for the considered datasets. However, the method could be recursively applied for deeper hierarchies.
- Orthogonal variant performance: While promising, the orthogonal variant ("Ortho") performed slightly worse than the "Sub" method on some datasets, indicating that the optimal approach might be dataset-dependent.
- Future work: The authors suggest exploring the use of these learned representations for tasks like computational creativity. They also note that while their learned spaces are "messier" than idealized conceptual spaces, their interpretability makes them valuable for bridging vector and symbolic representations.
Summary updated