A z-score measures how many standard deviations a data point is from the mean of a distribution. In data quality monitoring, z-scores compare a new run's column statistics against the historical baseline — flagging values that are statistically unusual.
Statistical anomaly detection in data quality is built on one foundational concept: the z-score. If you understand z-scores, you understand how systems like Collibra's Adaptive Rules, Monte Carlo's anomaly detection, and Sohovi's Behavioral Scoring actually work under the hood.
What Is a Z-Score?
The z-score formula is:
z = (x - μ) / σ
Where:
- x = the observed value (e.g., this run's null rate)
- μ (mu) = the mean of historical values
- σ (sigma) = the standard deviation of historical values
Sohovi profiles your datasets for quality issues in minutes — see what's broken before it breaks your pipeline — try Sohovi free.
A z-score of 0 means the observation is exactly at the historical mean. A z-score of 3 means the observation is 3 standard deviations above the mean. A z-score of -2 means it's 2 standard deviations below.
The sign tells you direction; the magnitude tells you how unusual the observation is.
Interpreting Z-Scores
Under a normal distribution, approximately:
- 68% of observations fall within 1 standard deviation (z between -1 and 1)
- 95% fall within 2 standard deviations (z between -2 and 2)
- 99.7% fall within 3 standard deviations (z between -3 and 3)
A z-score of 3 means there's only a 0.3% chance of seeing this value under normal conditions — it's statistically unusual. A z-score of 6 means a probability of about 1 in a billion.
This is why z ≥ 3 is the standard threshold for flagging anomalies in most systems. Values within ±3 sigma are considered "normal"; values outside that range warrant investigation.
Sohovi tracks quality trends across runs and alerts you when a metric — null rate, duplicate count, score — moves outside its normal range.
Applying Z-Scores to Data Quality Metrics
In data quality monitoring, you apply z-score analysis to column-level statistics across runs, not to individual data values. Specifically:
Null rate per column. If a column normally has 0.5% nulls (μ = 0.5, σ = 0.2), and a new run shows 8% nulls, the z-score is (8 - 0.5) / 0.2 = 37.5. That's a 37-sigma event — flagged as a critical anomaly.
Unique rate per column. If a column that should be unique normally has 99.8% unique values, and a run shows 85% unique values, the z-score is (85 - 99.8) / 0.1 = -148. Extreme duplication introduced.
Mean value for numeric columns. If revenue normally averages $180 with a std dev of $30, and a run shows an average of $890, the z-score is (890 - 180) / 30 = 23.7. Something unusual is happening with the revenue data.
Row count. If a weekly export normally produces 50,000 ± 2,000 rows, and a run produces 8,000 rows, the z-score is (8000 - 50000) / 2000 = -21. The pipeline is probably broken.
Why Z-Scores Work Well for This Problem
Z-scores are particularly well-suited for data quality monitoring for several reasons:
They're parameter-free for the user. You don't need to specify a range or threshold for each column individually. The threshold is automatically derived from the column's own historical behavior.
They're scale-invariant. A z-score of 3 means the same thing whether the column is measuring dollar amounts in millions or percentages between 0 and 1.
They adapt automatically. As more runs are recorded, the historical mean and std dev are updated. If the data legitimately shifts (new business conditions, seasonal patterns), the baseline adapts over time.
They quantify the degree of anomaly. Unlike a binary pass/fail, a z-score tells you how anomalous something is — a z-score of 3 is very different from a z-score of 30.
Limitations of Z-Score Detection
No method is perfect. Z-scores have specific limitations in the data quality context:
Requires sufficient history. With only 2–3 historical runs, the mean and std dev are noisy estimates. The baseline becomes more reliable with 10+ runs.
Assumes approximately normal distribution. Most time-series of data quality metrics do approximately follow normal distributions, but not always. For heavily skewed metrics, a z-score may over- or under-flag.
Doesn't detect slow drift. If a null rate increases by 0.1% per run over many months, no single run will have a high z-score — but the cumulative drift can be significant. Detecting gradual drift requires additional methods.
Generates false positives. With enough metrics being monitored, some will fall beyond 3 sigma by chance. In practice, high-severity anomalies (6+ sigma) are almost always genuine issues, while low-severity flags (3–4 sigma) sometimes represent legitimate variation.
Severity Levels in Practice
Most implementations use the z-score magnitude to set severity:
| Z-score range | Severity | Interpretation | |---|---|---| | 3.0 – 4.0 | Low | Unusual but possible by chance | | 4.0 – 6.0 | Medium | Statistically unlikely — warrants review | | 6.0+ | High | Extremely unusual — investigate immediately |
Sohovi uses the same banding: low = 3–4 sigma, medium = 4–6 sigma, high = 6+ sigma. Each severity level deducts a different number of points from the Behavior Score.
Key Takeaways
- Z-scores measure how many standard deviations an observation is from the historical mean
- In data quality, z-scores are applied to column-level statistics (null rate, mean, cardinality) across runs
- Values with |z| ≥ 3 are flagged as behavioral anomalies
- Z-scores require historical baseline data — more history = more reliable detection
- High z-scores (6+) almost always indicate genuine data quality issues
FAQ
Q: Do I need to calculate z-scores manually? A: No. Tools like Sohovi compute them automatically after each run and surface the results in the Behavior Score card.
Q: What's the difference between a z-score and a standard deviation? A: Standard deviation is a measure of spread in a distribution. Z-score measures a specific observation's distance from the mean in units of standard deviation.
Q: Can z-scores be negative? A: Yes. A negative z-score means the observed value is below the historical mean. For example, a run with fewer rows than usual has a negative z-score for row count.
Q: What happens when the standard deviation is zero? A: If every historical run had exactly the same value (σ = 0), any deviation from that value would technically be infinite. Systems typically treat this as an undefined z-score and handle it separately.
Q: How many historical runs are needed for reliable z-scores? A: Statistical convention suggests at least 30 observations for reliable estimates. In practice, 10 runs provide a reasonable baseline; 2 runs are the minimum for any detection.
Q: Is the 3-sigma threshold configurable? A: In Sohovi, it's set to 3 by default. Some enterprise tools allow users to configure per-column thresholds.
Q: What's the probability of a false positive at 3 sigma? A: Under a normal distribution, approximately 0.3% (about 1 in 370 observations). With many metrics monitored across many columns, false positives become more common.
Q: Can z-scores detect gradual drift? A: Not directly — z-scores detect sudden deviations. Gradual drift over many runs requires separate trend-detection methods.
Q: Why use z-scores rather than percentage change? A: Percentage change doesn't account for how variable a metric normally is. A 10% change is insignificant for a highly variable column but alarming for a stable one. Z-scores normalize for this.
Q: What if my data is highly seasonal? A: Seasonal patterns create legitimate high-z-score events at predictable intervals. For seasonal data, ideally compare against the same period in prior cycles rather than an undifferentiated rolling average.
