TL;DR
This paper presents a data-driven approach to automatically identify "Information Structure" (IS) in text, specifically the Topic and Focus of sentences. By training machine learning models on the Prague Dependency Treebank, they achieve over 90% accuracy, significantly outperforming a baseline and a rule-based system, and even surpassing human annotator agreement. This work demonstrates that syntactic features alone can effectively capture IS, opening doors for more nuanced NLP applications.
What Problem Does This Paper Address?
Understanding how information is structured within a sentence – what is already known or contextually bound (Topic) versus what is new or asserted (Focus) – is crucial for many Natural Language Processing (NLP) tasks. However, existing theories often rely on abstract principles and illustrative examples, making it difficult to implement them computationally. This paper tackles the challenge of automatically identifying this "Information Structure" (IS) in a data-driven way, moving beyond rule-based systems and manual annotation. They focus on the Praguian theory of Topic-Focus Articulation (TFA), which divides sentence content into Topic and Focus.
Methods
The authors employ machine learning techniques to automatically detect the TFA values (Topic/t or Focus/f) for each node in a tectogrammatical representation of a sentence. They experiment with three well-known classifiers:
- Decision Trees (C4.5): A popular algorithm for classification tasks.
- Maximum Entropy (MaxEnt): A statistical modeling approach.
- Rule Induction (RIPPER): An algorithm that learns a set of classification rules.
They utilize a set of 35 features, divided into two categories:
- Basic Features: Attributes directly extracted from the Prague Dependency Treebank (PDT) annotation.
- Derived Features: Features inspired by the annotation guidelines, computed using dependency information and surface word order.
The models are trained to predict whether a tectogrammatical node represents a Topic (t) or a Focus (f).
Datasets
The primary dataset used is the Prague Dependency Treebank (PDT). This corpus is unique as it's annotated with Information Structure following the Praguian approach of Topic-Focus Articulation. The authors use a portion of the PDT for training, development, and evaluation, with statistics provided in Table 3. They do not introduce a new benchmark dataset but rather leverage the existing rich annotation of the PDT.
Key Insights and Ideas
- Syntactic Features Suffice: The most striking insight is that a combination of basic and derived syntactic features, without explicit semantic or discourse-level information (beyond what's encoded in the treebank), is sufficient to achieve high accuracy in identifying Topic and Focus.
- Machine Learning Outperforms Rules: Data-driven machine learning approaches significantly outperform a rule-based system that directly implements the annotation guidelines. This highlights the limitations of hand-crafted rules when dealing with the nuances of IS.
- Surpassing Human Agreement: The best performing models achieve accuracy higher than the inter-annotator agreement for IS in the PDT. This suggests that the models might be learning a more consistent representation of IS than human annotators, or that the features capture subtle patterns that even annotators struggle with consistently.
- The Importance of Intonation Center (IC): The rule-based system's poor performance is attributed to the lack of explicit annotation for the "intonation center" (IC) in the corpus, a crucial factor in the manual annotation guidelines. This underscores the importance of prosodic information for IS.
- Learning Curves Show Early Saturation: The learning curves indicate that the models achieve high performance even with a small percentage of the training data, suggesting that the features are highly discriminative and that adding more data yields diminishing returns.
Key Results
- The best system (C4.5 classifier) achieved an accuracy of 90.69%.
- This represents a 44.73% improvement over the baseline system (which always assigns Focus), which achieved 62.66% accuracy.
- The rule-based system achieved 58.92% accuracy.
- The C4.5 model significantly outperformed MaxEnt (p < 0.001), and MaxEnt outperformed RIPPER (p < 0.01).
- The achieved accuracy (over 90%) is higher than the average inter-annotator agreement (86.42%).
Limitations and Future Work
- Intonation Center (IC): The lack of explicit IC annotation in the PDT is a significant limitation for rule-based systems and a potential area for improvement in feature engineering for ML models.
- Contrastive vs. Non-contrastive Topics: The current work does not distinguish between contrastive and non-contrastive Topic items, treating both as 't'. Future work could explore this distinction.
- Topic-Focus Partitioning of the Whole Sentence: The current work focuses on identifying Topic/Focus for individual nodes. The authors aim to extend this to determine the Topic-Focus partitioning of an entire sentence.
- English Corpus Development: They plan to leverage their experience with Czech data and parallel corpora to create an English corpus annotated with IS, transferring the identified topic/focus labels.
- Feature Set Improvement: Further investigation into improving the feature set is suggested.
Summary updated