The Difference Between Auditing and Monitoring
A data quality audit is a point-in-time assessment: you look at your data now and measure quality. An audit tells you what quality is today.
Data quality monitoring is continuous: automated checks run on a schedule, and alerts fire when metrics deviate from expected ranges. Monitoring tells you when quality changes.
Both are necessary. Audits set the baseline. Monitoring maintains it.
Sohovi tracks quality trends across runs and alerts you when a metric — null rate, duplicate count, score — moves outside its normal range.
What to Monitor
Volume anomalies: Sudden drops or spikes in record counts. A table that normally has ~10,000 new records daily shows 0 today — a data pipeline failure. A table shows 100,000 new records — a likely data import error.
Null rate changes: A column that was 98% complete last week is 70% complete this week. Something changed upstream: a form field was removed, a source system changed its schema, an import was corrupted.
Value distribution changes: The most common status value shifted dramatically. New values appeared in a field that should have fixed values. These signal schema or process changes upstream.
Freshness: A table that should be updated daily hasn't been updated in 36 hours. A data pipeline is silent when it should be active.
Cross-system consistency: Two systems that should agree on a value are diverging over time. A weekly check of a sample of records prevents silent consistency drift.
Setting Up Monitoring Without Enterprise Tools
Option 1: Scheduled SQL queries + email alerts Write SQL queries that calculate quality metrics. Schedule them to run daily (cron job or cloud scheduler). If results breach thresholds, send an email via a simple notification service.
Option 2: dbt tests on a schedule
Run dbt test as part of your daily data pipeline. Tests that fail generate notifications via Slack, PagerDuty, or email. Simple to implement if you're already using dbt.
Option 3: Great Expectations Define expectations programmatically. Run validation on each data load. Failed validations generate detailed reports with row-level examples.
Option 4: Purpose-built tools (Monte Carlo, Soda, Anomalo) These tools observe your data automatically, learn normal patterns, and alert on anomalies without you writing explicit rules. Useful for large pipelines where writing explicit tests for every potential issue isn't scalable.
Alert Design Principles
Alert on change, not just threshold: A metric that's always been at 85% completeness that suddenly drops to 80% deserves an alert — even though 80% might be "acceptable." The change is the signal.
Include context in alerts: "Email completeness dropped from 98% to 72%" is useful. "Data quality alert triggered" is not.
Route alerts to the right owner: The data owner for the affected table should receive the alert, not a general data team inbox. Unrouted alerts don't get actioned.
Avoid alert fatigue: Too many alerts means they all get ignored. Start with 5–10 critical metrics and expand only as alert handling processes are established.
