Abstract: We consider dynamic co-occurrence data, such as author-word links in papers published in successive years of the same conference. For static co-occurrence data, researchers often seek an embedding into a low-dimensional space. This paper extends this to dynamic data using a factored state-space model to track evolving relationships while maintaining temporal consistency.
Introduction
In data science, relationships are frequently represented as co-occurrence data—for example, which authors use specific keywords in research papers. Static embedding techniques like Multi-Dimensional Scaling (MDS) or Co-occurrence Data Embedding (CODE) map these entities into a low-dimensional space where proximity indicates a high degree of relation.
However, relationships are rarely static; research interests evolve, and new trends emerge. Attempting to embed snapshots of data year-by-year often results in noisy or inconsistent mappings, especially when data is sparse. The challenge is to create a model that maintains temporal consistency—leveraging historical data to inform the present—while remaining computationally efficient. This paper introduces D-CODE (Dynamic Co-occurrence Data Embedding) to address these needs.
The D-CODE Framework
D-CODE builds on the static CODE model by treating entity coordinates as hidden states in a Factored State-Space Model.
- The Latent Space: Every entity (e.g., an author $\phi$ or a word $\psi$) is assigned a coordinate in Euclidean space. The probability of co-occurrence is inversely proportional to the squared distance between them.
- The Dynamics: The model assumes coordinates follow a "random walk" over time (Gaussian diffusion). This acts as a temporal prior that prevents coordinates from jumping erratically between time steps.
- The Kalman Filter Approximation: Standard Kalman Filters require Gaussian distributions, but the original CODE likelihood is non-Gaussian. To maintain efficiency, the authors use Taylor expansions to approximate the observation model as a Gaussian. This allows for closed-form updates of the "belief" about an entity's location.
Experimental Evaluation
The model was tested on two primary datasets:
- Synthetic Data: A simulated sequence where two groups of authors ($X$ and $Y$) gradually swap their associations with a set of words ($A$).
- NIPS Corpus: A real-world dataset of word-author links from NIPS conference proceedings spanning 13 years (1986–1999), focusing on 40 prolific authors and 428 common words.
Results and Insights
A key innovation of D-CODE is that it provides a distribution rather than just point estimates. These "uncertainty ellipses" allow the model to capture the direction an entity is likely to move, which is vital for robust prediction. This approach also enables the model to "borrow strength" from adjacent years to produce reasonable embeddings even when data for a specific year is sparse.
Key Performance Metrics
- Author Prediction: In ranking which author likely wrote a paper based on keywords, D-CODE achieved a median rank of 4.1 (out of 20), significantly outperforming Dynamic PCA (rank 7) and Static CODE (rank 14.4).
- Qualitative Mapping: The model successfully clustered NIPS authors into recognizable sub-fields (e.g., Reinforcement Learning vs. Kernel Methods) and captured their movement as they transitioned between research topics over a decade.
- Gradual Adaptation: On synthetic data, D-CODE showed a smooth transition in predictions as trends shifted, whereas Maximum Likelihood (MLE) approaches were often overconfident and switched abruptly.
Conclusion and Future Work
While D-CODE provides a powerful framework for dynamic embedding, it has limitations. The Gaussian approximation used for the Kalman Filter is "uncontrolled," meaning the error relative to the true posterior is not formally bounded. Additionally, the precision matrix scaling may pose challenges for extremely large networks. Future research may explore Particle Filters or MCMC to handle more complex, bimodal posterior distributions where a single Gaussian is insufficient.
Summary updated