TakeLab Retriever: AI-Driven Search Engine for Articles from Croatian News Outlets

David Dukic, Marin Petricevic, Sven Curkovic, Jan SnajderarXiv 2024 · DOI · arXiv

Introduction

TakeLab Retriever is a specialized search engine designed to archive and semantically index the vast landscape of Croatian digital news. Spanning over two decades and encompassing more than 10 million articles, the system provides researchers with a transparent, unbiased alternative to general-purpose search engines for media analysis.

The Challenge of Media Analysis

For social scientists and media analysts, relying on mainstream search engines like Google or Bing presents significant hurdles. These platforms operate as "black boxes," often producing biased results that prioritize recent content over deep archives. Furthermore, they frequently struggle with the morphological complexities of lower-resource Slavic languages like Croatian.

General search engines also lack the granular semantic filtering required for rigorous academic study—such as distinguishing between "Tesla" the person and "Tesla" the company within a specific "Science" category. TakeLab Retriever was developed to bridge this gap, offering a tool tailored specifically to the Croatian media ecosystem.

System Architecture

The system is built on a robust microservice architecture managed via Docker, ensuring scalability and fault tolerance.

The TakeLab Retriever microservice architecture, including the scraper components and the multi-stage NLP pipeline.

The Scraper and Data Management

The scraper is a closed-loop system comprising a Scheduler, an asynchronous Downloader, and an Extractor. To manage the massive volume of data, the team employs SimHash, a locality-sensitive hashing technique, to deduplicate articles even when URLs or minor formatting details change. Inter-service communication is handled via PostgreSQL-based priority queues, allowing the system to balance the ingestion of new articles with the background reprocessing of archival content.

The NLP Pipeline

Articles are processed through a Directed Acyclic Graph (DAG) of specialized modules:

  • Core Module: Utilizes spaCy and the MOLEX lemmatizer for precise Croatian linguistic processing.
  • NER & NEL: Named Entity Recognition is performed by BERTić (a transformer model for South Slavic languages). Named Entity Linking (NEL) then maps these entities to Wikidata IDs using graph embeddings and PageRank scores.
  • Quality Control: An SVM classifier identifies "low-quality" content (e.g., horoscopes, photo galleries), which accounted for approximately 11.83% of the scraped data.
  • Topic Classification: The Omikuji library assigns articles to the international IPTC taxonomy (e.g., Politics, Economy, Sport).

The Living Archive

The engine's primary dataset is its own living archive, which includes over 10 million unique article URLs from 42 regional news outlets. By recursively following links within articles, the researchers uncovered a "hidden web" of archival content often missed by general crawlers. To ensure comprehensive coverage, the team even maintained subscriptions to paywalled outlets.

Semantic Search and Retrieval

TakeLab Retriever supports complex Boolean queries that allow researchers to perform high-precision searches across 33 major outlets simultaneously.

The search interface allows for granular filtering by combining entities, keywords, and IPTC topics.

By combining entity IDs, specific keywords, and standardized topics, users can isolate relevant data with high accuracy. For example, a query can specify the person "Nikola Tesla" (Wikidata Q9036) in conjunction with the keyword "magnet" within the "Science and Technology" topic.

Key Results and Performance

The system has successfully stabilized to process between 2,000 and 15,000 articles daily. The SimHash deduplication approach proved highly effective, yielding only two false positives in a test of 150,000 articles. This level of precision and scale makes the Retriever a powerful tool for longitudinal media studies.

Limitations and Future Directions

While the system is highly capable, some challenges remain. Data from the early 2000s is sparser due to the lack of inter-linking in older web structures. Future work will focus on implementing sentiment analysis, automated keyphrase extraction, and a user account system to facilitate reproducible research by allowing users to save and share search constraints.

Summary updated

Abstract

TakeLab Retriever is an AI-driven search engine designed to discover, collect, and semantically analyze news articles from Croatian news outlets. It offers a unique perspective on the history and current landscape of Croatian online news media, making it an essential tool for researchers seeking to uncover trends, patterns, and correlations that general-purpose search engines cannot provide. TakeLab retriever utilizes cutting-edge natural language processing (NLP) methods, enabling users to sift through articles using named entities, phrases, and topics through the web application. This technical report is divided into two parts: the first explains how TakeLab Retriever is utilized, while the second provides a detailed account of its design. In the second part, we also address the software engineering challenges involved and propose solutions for developing a microservice-based semantic search engine capable of handling over ten million news articles published over the past two decades.
BibTeX
@misc{dukic2024takelab,
  author = {David Dukic and Marin Petricevic and Sven Curkovic and Jan Snajder},
  title = {{TakeLab Retriever: AI-Driven Search Engine for Articles from Croatian News Outlets}},
  year = {2024},
  eprint = {2411.19718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.19718},
}
Figure