You're about to send a campaign to 15,000 contacts. Unknown to you, 2,800 of those contacts appear twice — some with the same email, some with slightly different names but the same address. The campaign goes out, 2,800 people receive it twice, and your unsubscribe rate spikes. All of this was preventable with a pre-send duplicate check.
Here's how to find duplicate records in a CSV file.
Step 1: Decide What "Duplicate" Means for Your Dataset
Not all duplicates are the same:
Sohovi automatically finds every duplicate in your dataset — including near-matches — and shows you exactly which rows are affected.
Exact duplicates — Identical values in all columns. The same row appears twice.
Key-field duplicates — Different rows but the same value in a key field (like email address). The same person with slightly different names or contact information.
Near-duplicates — "John Smith" and "Jon Smith" at the same company — probably the same person, but not an exact match.
Define which type you're looking for before you start.
Finding Exact Duplicates
In Excel: Select all data, go to Data > Remove Duplicates, check all columns. The dialog tells you how many duplicates were found before removing them.
In Google Sheets: Use Data > Remove Duplicates. Same behavior.
Finding Key-Field Duplicates (Same Email, Different Records)
In Excel:
- Add a helper column next to the email column
- Enter =COUNTIF($A:$A, A2) (where A is your email column)
- Any row where this formula returns >1 is a duplicate email
- Filter for values >1 to see all affected rows
Sorting alternative: Sort by email address. Duplicate emails will appear adjacently and are easy to spot visually.
Finding Near-Duplicates
Near-duplicates ("John Smith" vs. "Jon Smith") require fuzzy matching, which Excel doesn't do natively. Options:
- Manual review: Sort by last name, then first name. Similar names will cluster together.
- Fuzzy matching tools: Sohovi and dedicated deduplication tools use string similarity algorithms to identify near-matches automatically.
After Finding Duplicates
Count them before merging or deleting. Note the percentage (if 15% of your records are duplicates, that's a significant finding that needs a source investigation).
For exact duplicates: delete all but the most complete/recent record.
For key-field duplicates: review each pair manually and merge into a single record, combining information from both.
For near-duplicates: higher risk — review carefully before merging to avoid incorrectly merging two distinct people.
Build deduplication into your import process going forward, not just your cleanup process.
