Abstract
On the basis of the text extraction based on the density of text, the Web page text extraction algorithm based on the adaptive threshold was proposed and applied in the new rural community employment information service system for the employment information fetching from the related government affairs website combined with the Otsu threshold algorithm. Through the web page text extraction contrast experiments to the Webpages including "The ministry of human resources and social security of the People's Republic of China", "The ministry of human resources and social security hall of henan province" and "Sina.com", the text extraction rate of the algorithm reached 90%, 92% and 92% respectively. The results showed that the application of the algorithm in new rural community employment information service system could provide technical support for the directional employment information acquisition and realize accurate employment information retrieval.
Introduction
Web scraping is rarely as simple as "get all text." Most web pages are cluttered with "noise": navigation menus, copyright footers, advertising links, and JavaScript snippets. For a system designed to aggregate employment opportunities for rural communities, missing the actual job description or accidentally including a sidebar ad for a different service is a failure.
The standard approach uses text density—the ratio of actual text to HTML tags in a given block. However, every website has a different "cleanliness" level. Traditionally, developers had to manually set a threshold value for each site to decide what counts as "main body text." This doesn't scale. The authors sought a way to make this threshold adaptive, allowing the scraper to adjust itself to the specific layout of any given page.
Methodology: Adaptive Thresholding via Otsu's Method
The proposed system operates in two primary stages:
- Text Density Calculation: The algorithm cleans the HTML (removing scripts and tags while preserving line breaks) and groups lines into "Row Blocks" (typically 5 lines) to calculate the B-length (the number of non-blank characters).
- Adaptive Thresholding: This is the core innovation. The Otsu threshold algorithm, a classic computer vision tool used to separate an image's foreground from its background, is applied to the 1D distribution of text density. By treating "text areas" and "non-text areas" as two distinct classes, the algorithm automatically finds the optimal threshold to isolate the article body.
The system was integrated into a "New Rural Community Employment Information Service System," which follows a workflow of extracting column headers, then titles, and finally the main text body using the adaptive threshold.
Key Insights and Results
The most significant innovation is the application of Otsu’s method—traditionally used for pixel intensities—to the "intensity" of text density. This treats a web page's structure like a signal processing problem. The algorithm looks for the transition points: the main body of an article usually starts with a sudden surge in text characters and ends with a sharp drop-off, which is a more robust indicator than a static density value.
In testing across three real-world sources, the adaptive algorithm achieved high extraction rates:
- Government Sites: On the "cleaner" government sites (National and Henan regional), the algorithm successfully captured the full text in 90% to 92% of cases.
- Commercial Sites: On the messier Sina.com, the success rate dropped to 64%, though this still outperformed basic static-threshold algorithms (58%).
Conclusion
While the adaptive algorithm is significantly more computationally expensive—taking roughly 4 to 5 times longer than simpler methods—it provides a robust solution for automated content extraction without manual configuration. Future research will focus on improving the efficiency of the calculation to reduce fetching time without sacrificing the accuracy gains provided by the adaptive threshold.
Summary updated