Introduction
TakeLab Retriever is a specialized search engine designed to archive and semantically index the vast landscape of Croatian digital news. Spanning over two decades and encompassing more than 10 million articles, the system provides researchers with a transparent, unbiased alternative to general-purpose search engines for media analysis.
The Challenge of Media Analysis
For social scientists and media analysts, relying on mainstream search engines like Google or Bing presents significant hurdles. These platforms operate as "black boxes," often producing biased results that prioritize recent content over deep archives. Furthermore, they frequently struggle with the morphological complexities of lower-resource Slavic languages like Croatian.
General search engines also lack the granular semantic filtering required for rigorous academic study—such as distinguishing between "Tesla" the person and "Tesla" the company within a specific "Science" category. TakeLab Retriever was developed to bridge this gap, offering a tool tailored specifically to the Croatian media ecosystem.
System Architecture
The system is built on a robust microservice architecture managed via Docker, ensuring scalability and fault tolerance.
The Scraper and Data Management
The scraper is a closed-loop system comprising a Scheduler, an asynchronous Downloader, and an Extractor. To manage the massive volume of data, the team employs SimHash, a locality-sensitive hashing technique, to deduplicate articles even when URLs or minor formatting details change. Inter-service communication is handled via PostgreSQL-based priority queues, allowing the system to balance the ingestion of new articles with the background reprocessing of archival content.
The NLP Pipeline
Articles are processed through a Directed Acyclic Graph (DAG) of specialized modules:
- Core Module: Utilizes
spaCyand the MOLEX lemmatizer for precise Croatian linguistic processing. - NER & NEL: Named Entity Recognition is performed by BERTić (a transformer model for South Slavic languages). Named Entity Linking (NEL) then maps these entities to Wikidata IDs using graph embeddings and PageRank scores.
- Quality Control: An SVM classifier identifies "low-quality" content (e.g., horoscopes, photo galleries), which accounted for approximately 11.83% of the scraped data.
- Topic Classification: The Omikuji library assigns articles to the international IPTC taxonomy (e.g., Politics, Economy, Sport).
The Living Archive
The engine's primary dataset is its own living archive, which includes over 10 million unique article URLs from 42 regional news outlets. By recursively following links within articles, the researchers uncovered a "hidden web" of archival content often missed by general crawlers. To ensure comprehensive coverage, the team even maintained subscriptions to paywalled outlets.
Semantic Search and Retrieval
TakeLab Retriever supports complex Boolean queries that allow researchers to perform high-precision searches across 33 major outlets simultaneously.
By combining entity IDs, specific keywords, and standardized topics, users can isolate relevant data with high accuracy. For example, a query can specify the person "Nikola Tesla" (Wikidata Q9036) in conjunction with the keyword "magnet" within the "Science and Technology" topic.
Key Results and Performance
The system has successfully stabilized to process between 2,000 and 15,000 articles daily. The SimHash deduplication approach proved highly effective, yielding only two false positives in a test of 150,000 articles. This level of precision and scale makes the Retriever a powerful tool for longitudinal media studies.
Limitations and Future Directions
While the system is highly capable, some challenges remain. Data from the early 2000s is sparser due to the lack of inter-linking in older web structures. Future work will focus on implementing sentiment analysis, automated keyphrase extraction, and a user account system to facilitate reproducible research by allowing users to save and share search constraints.
Summary updated