You have a CSV file you need to audit. You know enterprise profiling tools exist — IBM, Informatica, Talend — but they're priced for data engineering teams and require days of setup. You need a quality check on this file today. Here's how to do it without enterprise software.
What You're Trying to Learn
Before choosing a method, clarify what you need to know about the CSV:
- Which columns are mostly empty (completeness)?
- Are there duplicate rows or duplicate values in key fields (uniqueness)?
- Do columns have consistent formats (validity and conformity)?
- What are the most common values (distribution)?
- Does the file contain personal data (PII)?
Sohovi automatically finds every duplicate in your dataset — including near-matches — and shows you exactly which rows are affected.
The method you choose depends on how much of this you need and how quickly.
Option 1: Sohovi (Fastest, No Setup)
Upload your CSV to Sohovi and get an instant profile of every column — completeness rates, distinct value counts, format patterns, uniqueness scores, and PII detection — entirely in your browser. Your file never leaves your machine. No account required for a basic profile.
This is the fastest option for non-technical users and for any CSV under a few hundred thousand rows.
Option 2: Excel or Google Sheets (Manual, No Additional Software)
For a small CSV (under 50,000 rows):
- Completeness: Use COUNTBLANK() to count empty cells per column
- Duplicates: Use Remove Duplicates or COUNTIF to find repeated values
- Distribution: Use COUNTIF or a pivot table to see value frequencies
- Min/Max: Use MIN() and MAX() on numeric columns
This works but is time-consuming and doesn't scale to large files.
Option 3: Python (Powerful, Requires Basic Coding)
The pandas library makes CSV profiling straightforward:
- df.info() — column names, types, non-null counts
- df.describe() — statistics for numeric columns
- df.nunique() — distinct value counts per column
- df.duplicated().sum() — duplicate row count
If you're comfortable with Python, this is powerful and flexible.
What to Look For in the Profile
Once you have your profile output, focus on:
- Any column with completeness below 80% (or 100% for key fields)
- Any column where you expected uniqueness but found duplicates
- Any numeric column with unexpected min/max values (outliers or system defaults)
- Any categorical column with far more distinct values than expected
- Any column with mixed data types (some numeric, some text)
See our guide to what data profiling reveals in practice for more detail on interpreting results.
The goal of profiling isn't perfection — it's visibility. Once you know what's in the file, you can decide what to fix and what's acceptable for your specific use case.
