TL;DR
This paper introduces an unsupervised system for extracting structured information from short, informal online listings (like classified ads). It tackles the challenge of noisy, abbreviated text by building a semantic model that identifies main objects and their attributes, improving upon traditional methods that struggle with this type of data.
What Problem Does This Paper Address?
The internet is awash with short, informal texts like classified ads and product listings. These listings, often just a few words long, are incredibly valuable for tasks like market analysis, faceted search, and automated reasoning. However, their informal nature – replete with typos, abbreviations, and unconventional phrasing – makes them a nightmare for traditional Information Extraction (IE) systems designed for formal text. Existing IE techniques often require extensive human supervision to build knowledge bases or annotate training data, which is costly and time-consuming, especially when dealing with new domains. This paper aims to bridge this gap by developing an unsupervised IE system specifically tailored for these challenging short listings.
Methods
The core of the proposed system lies in building a "lightweight semantic model" without human supervision. This model captures two key pieces of information: the main concept (e.g., "TV," "condo") and its attributes (e.g., "size," "brand," "bedroom number"). The system employs two main components to achieve this:
- Shallow Semantic Parser (SSP): This component first preprocesses the listings, handling issues like duplicate entries and HTML tags. It then performs multiword detection (e.g., identifying "granite top" as a single unit) and tokenization heuristics to deal with abbreviations and mixed-case words. Crucially, it identifies potential main concepts and their attributes within each listing.
- Listing-Focused Clustering (LFC): After the SSP provides initial candidates, the LFC module groups these tokens into meaningful clusters. It defines two types of attributes: quantitative (e.g., "2 bedroom") and qualitative (e.g., "washer hookup"). LFC uses a combination of context-based similarity and a set of custom rules designed for short listings to group tokens of the same type. These rules address issues like preventing quantitative attributes that frequently co-occur (like "bed" and "bath") from being clustered together, and ensuring quantitative and qualitative attributes are not mixed.
The system uses a bootstrapped approach, iteratively refining its understanding of main concepts and attributes.
Datasets
The authors evaluated their system on three distinct datasets:
- Craigslist (Rent Ad): Housing rental advertisements.
- EBay (Glove): Auction listings for baseball gloves.
- FatWallet (TV Deal): Hot deal postings for TVs and projectors.
These datasets were used for both training (implicitly, as the system is unsupervised) and testing. The paper also mentions a development dataset used for parameter tuning.
Key Insights and Ideas
- Informal text is a distinct challenge: Traditional IE methods fail on short, informal listings due to their unique linguistic characteristics.
- Unsupervised IE is feasible and valuable: The paper demonstrates that it's possible to build effective IE systems without human annotation for these challenging texts.
- Semantic model is key: Representing listings by their main concept and attributes provides a structured way to approach extraction.
- Custom clustering rules are vital: Standard context-based clustering isn't enough for short listings; specialized rules are needed to handle their specific properties.
- Main concepts are easier to extract: The system achieves higher F1-scores for main concept extraction (like "housing_type" or "product_type") compared to some qualitative attributes, highlighting the effectiveness of the semantic parser.
- Qualitative attributes are inherently harder: Lower inter-annotator agreement for qualitative attributes like "location" or "neighborhood" suggests these are inherently more ambiguous and difficult to extract consistently, even for humans.
Key Results
The system achieved F1-scores ranging from 0.35 to 0.66 across the three domains for the full extraction task (exact match).
- Rent Ad: F1-score of 0.35 (considered more difficult).
- Glove: F1-score of 0.54.
- TV Deal: F1-score of 0.70.
The paper also shows that the shallow semantic parser (SSP) and listing-focused clustering (LFC) components significantly improve performance over a baseline system that only uses space-based tokenization and context-based clustering. The full system (SSP + LFC) outperformed baselines and individual component additions across most metrics.
Limitations and Future Work
- Fine-grained clusters: The system's clusters can sometimes be too fine-grained, leading to missed information types ("NOT IN MAPPING" errors).
- Tokenization of qualitative attributes: Improving tokenization for qualitative attributes like "neighborhood" or "display technology" is suggested.
- More informative tokens: The system could benefit from extracting more informative tokens, especially for the "Rent Ad" dataset.
Future work includes:
- Comparing with supervised systems: To quantify the gap between unsupervised and supervised approaches.
- Joint learning: Exploring a unified approach that jointly learns tokenization, main concept clustering, and attribute clustering.
- External knowledge resources: Investigating the use of external knowledge bases like DBPedia and FreeBase to further enhance performance.
Summary updated