Entity Resolution (ER)—the process of identifying different records that refer to the same real-world object—is a foundational challenge in data integration. Historically, ER algorithms operated under a comfortable assumption: that the datasets being compared shared a clean, well-defined schema. If you wanted to deduplicate a customer database, you simply aligned the First_Name and Last_Name columns and computed text similarity scores.
But as data has migrated to web-scale data lakes, this assumption has completely broken down. Today, data integration must happen across highly heterogeneous information spaces. An algorithm might need to match a row in a relational database to a semi-structured JSON snippet, an RDF graph, or a string of unstructured text.
As traced by Papadakis et al. in their historical overview, this extreme variety introduces thousands of inconsistent, missing, or non-existent attribute labels. For instance, web-scale product integration often requires reconciling data from thousands of independent e-commerce sites, resulting in a massive explosion of synonymous but differently named attributes for the exact same entity type (Barbosa et al. Big Data Integration for Product Specifications). Even when data appears structured, such as in web tables, the underlying HTML and attribute order can vary wildly from site to site (Kawamura et al. HTML-LSTM: Information Extraction from HTML Tables in Web Pages Using Tree-Structured LSTM).
To solve this, a powerful paradigm shift has emerged: schema-agnostic entity resolution. Instead of trying to fix or align schemas before matching, these techniques ignore the schema entirely, treating records as unstructured collections of tokens.
The Core Intuition: Values Over Metadata
Why can we afford to ignore schemas? The core intuition is that the identity of an entity is carried by its values, not its column names.
Consider two records representing the same book:
- Record A (Relational):
[Title: "Introduction to Algorithms", Author: "Cormen", Year: "2009"] - Record B (JSON):
{ "book_name": "Introduction to Algorithms", "written_by": "Cormen" }
Traditional ER spends significant computational effort trying to determine that Title maps to book_name and Author maps to written_by. Schema-agnostic ER bypasses this step entirely. It flattens both records into simple "bags of tokens":
- Tokens A:
{"Introduction", "to", "Algorithms", "Cormen", "2009"} - Tokens B:
{"Introduction", "to", "Algorithms", "Cormen"}
By shifting the focus from where a piece of data lives (metadata) to what the data actually says (values), we can perform entity resolution across completely different formats. As Simonini et al. demonstrate, treating records purely as token collections allows a single progressive framework to seamlessly ingest and process RDF, JSON, and CSV files simultaneously without any prior structural alignment (Simonini et al.).
Taming the Quadratic Nightmare: Attribute-Agnostic Blocking
Naively comparing every bag of tokens against every other bag creates a "quadratic nightmare" ($O(N^2)$ complexity), which is completely unfeasible for millions of web entities. Traditional ER uses "blocking" to group similar records based on specific attribute values (e.g., placing all records with the same ZipCode into the same bucket). Without a schema, we cannot target specific attributes.
To overcome this, Papadakis et al. pioneered attribute-agnostic blocking (Papadakis et al.). Instead of relying on a single key, the system extracts tokens from every available part of an entity—including URI segments, literals, and related identifiers—and creates a block for every token. If two entities share a token, they land in the same block.
Record 1: [Name: "Apple iPhone 13", Vendor: "BestBuy"] --> Blocks: [Apple], [iPhone], [13], [BestBuy]
Record 2: { "prod": "iPhone 13", "brand": "Apple" } --> Blocks: [iPhone], [13], [Apple]
While this ensures an exceptionally high recall (matching entities will almost certainly share at least one token block), it creates a massive number of redundant blocks and unnecessary comparisons. To make this practical, schema-agnostic frameworks employ two main optimization strategies:
- High-Value Token Selection: Not all tokens are created equal. Common words like "and" or "com" create massive, useless blocks. O'Hare et al. introduced High-Value Token-Blocking (HVTB), which applies TF-IDF to the tokenized records. By only blocking on the most distinctive, "high-value" tokens, the system can efficiently link highly disparate formats—such as matching a short document title to a long, unstructured abstract paragraph—while drastically reducing memory usage.
- Comparison Cleaning Layers: In their robust two-layer framework, Papadakis et al. decoupled block generation from comparison execution (Papadakis et al.). An efficiency layer runs meta-blocking algorithms that track how many blocks two entities share. If two records only overlap on a single, highly frequent block, the comparison is purged. This allows the system to scale to over 100 million entities, executing as few as 13 targeted comparisons per entity.
Generalized Entity Matching
Once candidate records are grouped into blocks, the system must make a final decision: do these two records actually represent the same entity? This phase is known as Generalized Entity Matching (GEM).
Because schemas cannot be relied upon to align features, GEM algorithms rely heavily on the statistical distribution of tokens and structural graph relationships. For example, the MinoanER framework developed by Efthymiou et al. uses schema-agnostic similarity metrics that evaluate both the token overlap of the entities and the similarity of their neighboring entities in a data graph (Efthymiou et al.; Papadakis et al.). This allows the system to resolve entities across diverse knowledge bases without requiring manual attribute mapping.
To scale this across the web, Kejriwal proposed an unsupervised, cloud-deployable architecture that handles "N-Way" matching (Kejriwal). Instead of performing isolated pairwise comparisons between two datasets, the pipeline ingests an arbitrary number of heterogeneous data models (such as RDF graphs and relational tables simultaneously) and resolves entities globally based on emergent token patterns.
The Role of Language Models
The rise of deep learning has pushed GEM even further. Traditional token matching can fail when faced with synonyms or slight typographical variations. To evaluate how well modern architectures handle these structural mismatches, Wang et al. established the Machamp benchmark (Wang et al.), explicitly challenging models to match entities across mismatched relational, JSON, and textual formats.
To tackle this, recent frameworks leverage the semantic understanding of Pre-trained Language Models (PLMs). Because PLMs process text sequentially, a schema-agnostic record can simply be serialized into a flat string of text.
- Wang et al. introduced PromptEM (Wang et al.), which uses prompt-tuning and uncertainty-aware self-training to achieve high matching accuracy across formats even when labeled training data is scarce.
- Zhang et al. demonstrated that small language models can perform efficient, zero-shot entity matching (Zhang et al.). Because the language model understands the underlying semantics of the words rather than relying on rigid column headers, it can accurately determine if a JSON object and a relational row match, even when metadata is completely missing or corrupted.
Conclusion
Schema-agnostic entity resolution works because it embraces the inherent messiness of web-scale data rather than trying to fix it. By treating records as flat collections of tokens, utilizing attribute-agnostic blocking to prune the search space, and leveraging semantic or statistical matching models, it provides a scalable, hands-off solution for integrating the world's most diverse data sources.
Key Foundational Work
Beyond 100 million entities: large-scale blocking-based resolution for heterogeneous data (2012) — contributed 2 concept elements to this direction.
A Blocking Framework for Entity Resolution in Highly Heterogeneous Information Spaces (2013) — contributed 2 concept elements to this direction.
Discovering Linkage Points over Web Data (2013) — contributed 1 concept elements to this direction.