Discourse parsing—the task of identifying how units of text like clauses or sentences relate to one another—is fundamental to understanding document coherence. Traditionally, this task relies on Rhetorical Structure Theory (RST), which represents documents as hierarchical trees. However, supervised discourse parsing faces significant hurdles: expert annotation is prohibitively expensive, leading to small datasets (like the 385-article RST-DT) that suffer from severe domain bias.
To address these challenges, this paper introduces the Tree Autoencoder (T-AE), a novel framework for unsupervised discourse induction. By leveraging a self-supervised reconstruction objective, the T-AE learns to discover the "skeleton" of a text without requiring human-labeled trees.
The Tree Autoencoder Architecture
The core premise of the T-AE is that if a model can compress a document into a latent vector and then accurately reconstruct its original components, it must have captured the underlying structural relationships between those components.
Encoder and Decoder
The Encoder utilizes a TreeLSTM architecture to process adjacent Elementary Discourse Units (EDUs). It makes discrete "merge" decisions to build a tree structure. To ensure the model remains end-to-end differentiable, the authors employ the Gumbel-Softmax trick for these selection steps.
The Decoder performs the inverse operation. Starting from the final document-level vector, it recursively splits the representation back into the original EDU embeddings, following the tree structure identified during encoding. The model is trained to minimize the Mean-Squared Error (MSE) between the original and reconstructed text units.
Experimental Setup and Datasets
The researchers evaluated the T-AE using two primary datasets:
- RST-DT: The standard benchmark for discourse parsing, used here to evaluate how well unsupervised trees align with human gold standards.
- Yelp’13: A large-scale dataset of approximately 335,000 restaurant reviews used for unsupervised training.
By training on the massive Yelp dataset, the authors aimed to demonstrate that data scale can compensate for the lack of explicit linguistic supervision.
Key Results and Insights
The T-AE demonstrates remarkable performance for an unsupervised model. When trained on Yelp reviews, it achieved a 71.32% micro-precision score on the RST-DT benchmark, significantly outperforming traditional baselines such as Left-Branching (53.73%) and even some linguistically-informed hierarchical models.
A critical finding was that the model trained on Yelp reviews outperformed a version trained directly on the target domain (RST-DT). This suggests that the "grammar" of discourse—the way arguments and narratives are structured—is somewhat universal and that large-scale unsupervised learning can capture these patterns more effectively than small-scale supervised learning.
Furthermore, the T-AE is task-agnostic. Unlike previous "latent tree" models that optimize for a specific downstream task like sentiment analysis, the T-AE learns a general structural representation. Despite this, the resulting document vectors proved highly effective for sentiment classification, indicating that structural understanding naturally aids semantic tasks.
Conclusion and Future Directions
The Tree Autoencoder represents a significant step toward scalable discourse parsing. While it currently focuses on tree structure rather than relationship labeling (e.g., "Contrast" vs. "Elaboration"), its ability to generalize across domains without human intervention is a major advantage. Future work may involve integrating Variational Autoencoders (VAEs) or pre-training strategies to further enhance the model's generative and representative capabilities.
Summary updated