A Novel Method for Bilingual Web Page Acquisition from Search Engine Web Records

Yanhui Feng, Yu Hong, Zhenxiang Yan, Jian-Min Yao, Qiaoming ZhuCOLING 2010 · OpenReview

Abstract

Building large-scale parallel corpora is a fundamental challenge in machine translation. This paper proposes a novel method to acquire bilingual web pages by exploiting the structured data provided by search engines. The system operates in two stages: first, it automatically extracts web records from search engine result pages (SERPs) using tag path clustering; second, it employs a Support Vector Machine (SVM) to verify the bilingual nature of the linked pages using snippet-based features. Evaluation across six major search engines shows that the method achieves a record extraction precision of 99% and a bilingual verification recall of nearly 95%, offering a highly efficient alternative to traditional web crawling.

Introduction

The performance of statistical machine translation (SMT) systems is heavily dependent on the availability of high-quality parallel text. While traditional methods focus on crawling websites with symmetric URL structures (e.g., English and Chinese versions of the same site), they often overlook "mixed" bilingual pages where both languages appear on a single page. These pages are abundant but difficult to locate via standard crawling. This research leverages the indexing power of commercial search engines to identify these pages through their result snippets, significantly reducing the computational overhead of data acquisition.

System Architecture

The proposed pipeline consists of several modules designed to transform raw internet search results into verified bilingual web pages. The process begins with submitting parallel sentence pairs to search engines and ends with the classification of the resulting web records.

The overall architecture of the bilingual web page acquisition system.

Methodology

Web Record Extraction

Search engine result pages are highly structured, containing multiple records with titles, snippets, and URLs. To extract these without manual rule-writing for every engine, the authors utilize Tag Path analysis. A tag path represents the hierarchical HTML structure of a data element. By applying Affinity Propagation (AP) clustering to these paths on sample pages, the system automatically learns the "wrapper" for any search engine. This allows for the precise extraction of individual result records with minimal manual intervention, adapting to new search engine layouts in under 30 seconds.

Bilingual Verification via Snippets

Once records are extracted, the system must determine if the target page is bilingual. Rather than downloading the full page, the method analyzes the search snippet—the query-dependent summary provided by the search engine.

Example of search result records for a bilingual query on Baidu, showing the snippets used for verification.

The verification process uses a Support Vector Machine (SVM) classifier trained on three feature sets:

  1. Word-Overlap: Identifying translation equivalents within the snippet using a bilingual dictionary.
  2. Length-Based Features: Comparing the relative lengths of English and Chinese segments to ensure proportionality.
  3. Frequency-Based Features: Measuring the occurrence of query terms within the snippet text.

Experimental Results

The method was evaluated on a dataset of 2,516 records automatically extracted from six major search engines: Google, Baidu, Yahoo, Youdao, Bing, and Tecent Soso. The record extraction module proved highly robust, achieving 99% precision and 98% recall.

In the bilingual verification stage, the system reached an average precision of 81.3% and a recall of 94.93% when evaluated against snippet-based annotations. The study found that adding structural features (length and frequency) to the basic word-overlap measure improved precision from 70.2% to 85.1%, demonstrating that metadata is as vital as semantic content in this context.

Conclusion and Future Work

This research demonstrates that search engine snippets are effective proxies for identifying bilingual web content. By automating the wrapper generation process and utilizing lightweight snippet features, the proposed method provides a scalable and efficient way to expand parallel corpora for NLP. Future work will address the challenges of snippet quality—where search engine summaries may not perfectly reflect page content—and explore more diverse strategies for selecting initial parallel seed queries.

Summary updated

BibTeX
@inproceedings{feng2010novel,
  author = {Yanhui Feng and Yu Hong and Zhenxiang Yan and Jian-Min Yao and Qiaoming Zhu},
  title = {{A Novel Method for Bilingual Web Page Acquisition from Search Engine Web Records}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=ioO5nWeRdK},
}
Figure