Record linkage is the process of identifying records that refer to the same real-world entity across different datasets, even when those datasets don't share a common unique identifier — relying on combinations of partially identifying fields (name, address, date of birth) to establish a match.
Most deduplication is straightforward: find records in the same system that share the same email address. Record linkage is different — it's used when you need to match records across systems that have no common unique key, or when the data was collected at different times under different conditions and the only information linking two records is approximate.
When You Need Record Linkage
Merging data from two organizations: You've acquired a company. They have 50,000 customers. You have 80,000. Some are the same people. Neither system has the other's customer IDs. Name + address + phone is the only basis for matching.
Linking historical records without modern identifiers: You're trying to connect old paper records that were digitized with modern database records. The old records have name and address; the modern records have email and phone. No common key exists.
Connecting records across public and private datasets: You need to connect your customer database with a government registry or public health dataset. Name + date of birth + address is the only shared information.
See exactly what's wrong with your data — try Sohovi free — try Sohovi free.
Deduplicating records from multiple forms with no shared ID: A conference collected attendee information across four registration channels. Each collected slightly different fields. You need to identify which records across channels represent the same person.
Record Linkage vs. Deduplication vs. Entity Resolution
These three terms are related but distinct:
Deduplication: Finding and merging duplicate records within a single dataset. The same entity appears twice in the same table.
Sohovi automatically finds every duplicate in your dataset — including near-matches — and shows you exactly which rows are affected.
Record linkage: Finding matching records across two or more different datasets. Entity A in Dataset 1 corresponds to Entity B in Dataset 2.
Entity resolution: The broader process of determining which records refer to the same real-world entity, regardless of whether they're in the same dataset or different ones. Encompasses both deduplication and record linkage.
How Record Linkage Works
Step 1: Define the comparison variables. Which fields will be compared? Name similarity, address similarity, date of birth, phone number, geographic proximity. The combination of variables determines matching power.
Step 2: Apply blocking. Reduce the comparison space by grouping records that share some common attribute (first letter of last name, ZIP code, birth year). Only compare records in the same block.
Step 3: Calculate similarity scores. For each pair of records in the same block, calculate a composite similarity score across all comparison variables. Fuzzy matching algorithms handle each field type.
Step 4: Apply threshold classification. Records above the high threshold are classified as matches. Records below the low threshold are classified as non-matches. Records between the thresholds go to a manual review queue.
Sohovi tracks quality trends across runs and alerts you when a metric — null rate, duplicate count, score — moves outside its normal range.
Step 5: Human review for uncertain cases. A trained reviewer examines borderline pairs and makes a final match/non-match decision. This human judgment is what makes record linkage accurate in ambiguous cases.
Frequently Asked Questions
Q: What is record linkage in data quality? Record linkage is the process of identifying records that refer to the same real-world entity across different datasets that lack a common unique identifier. It uses combinations of partially identifying fields (name, address, date of birth) and similarity scoring to establish matches.
Q: What's the difference between record linkage and deduplication? Deduplication finds and removes duplicate records within a single dataset. Record linkage identifies corresponding records across two or more different datasets. Both use similar matching techniques; they differ in scope — deduplication is within-dataset, record linkage is cross-dataset.
Q: What is entity resolution? Entity resolution (also called entity matching or identity resolution) is the broader process of determining which records refer to the same real-world entity, whether within one dataset or across multiple datasets. It encompasses both deduplication and record linkage.
Q: What fields are most useful for record linkage? The most discriminating fields are those that are both unique to individuals and reliably captured: full name (first + last), date of birth, current residential address, and phone number. Social security numbers and other national IDs are highly unique but often unavailable. No single field is sufficient; record linkage relies on combinations.
Q: How is record linkage different from fuzzy matching? Fuzzy matching is one technique used within record linkage — it measures string similarity between field values. Record linkage is the overall process that uses fuzzy matching (and other similarity measures) on multiple fields to produce a composite match decision. Record linkage includes blocking, multi-field scoring, and threshold classification; fuzzy matching is one component.
Q: What is deterministic vs. probabilistic record linkage? Deterministic (rule-based) linkage uses explicit rules: "records match if last name is identical and date of birth is identical and ZIP code is within 10 miles." Probabilistic linkage assigns probability scores to matches based on how often each field combination would be expected to match by chance. Probabilistic linkage is more powerful for noisy data; deterministic linkage is simpler to implement and explain.
Q: When is human review essential in record linkage? Human review is essential for records that fall in the uncertain zone — neither clearly a match nor clearly distinct. The percentage requiring human review depends on the quality of the data and the threshold settings, but in most practical record linkage projects, 5–20% of pairs require review.
Q: What is the Fellegi-Sunter model? The Fellegi-Sunter model is a foundational probabilistic record linkage framework that formalizes the decision process: for each candidate pair, calculate the likelihood ratio that the pair represents a true match vs. a coincidental match. It provides a principled basis for setting match/non-match thresholds.
Q: What tools support record linkage? Python libraries (recordlinkage, splink, dedupe) support probabilistic record linkage. Commercial identity resolution platforms (LiveRamp, Informatica MDM, Ataccama) provide enterprise-grade capabilities. For simpler cross-dataset matching (two systems linked by email), a spreadsheet VLOOKUP is sufficient.
Q: How accurate can record linkage be? Recall and precision both in the 90–95% range are achievable on clean, moderately sized datasets with good field coverage. Accuracy drops with data quality — missing fields, spelling variations, and outdated information all reduce match confidence. Perfect accuracy is not achievable for large-scale probabilistic record linkage.
Record linkage is the tool for the hard matching problems that standard deduplication can't solve. When you need to connect data across systems that share no common key, it's the right approach — though it requires more careful design and human review than simpler deduplication.
