A behavioral anomaly is a data quality issue that can only be detected by comparing current data statistics against historical baselines — it's a deviation from normal behavior, not a violation of a fixed rule.
Most data quality tools focus on rule violations: a field is null when it shouldn't be, a value is outside a range, a format doesn't match a pattern. These are absolute constraints that hold regardless of context.
Behavioral anomalies are different. They're relative: something that looks fine in isolation but is statistically unusual given the history of this particular dataset.
The Classic Example
Imagine a customer database that normally has 0.3% null email addresses. This week's export has 14% null emails. Your null check rule has a threshold of 50% — so it passes. The DQ score is fine. But something has clearly gone wrong: the null rate jumped by 46x.
Sohovi tracks quality trends across runs and alerts you when a metric — null rate, duplicate count, score — moves outside its normal range.
A behavioral anomaly detector would catch this because it knows the historical baseline: 0.3% is normal; 14% is a 40-sigma event. That's not a coincidence — it's a signal.
How Behavioral Anomaly Detection Works
The algorithm has three steps:
1. Build a baseline. After each run, store the per-column summary statistics: null rate, unique rate, mean, standard deviation, min, max, dominant values. Over multiple runs, this forms a historical distribution for each metric.
2. Compare new runs against the baseline. When a new run completes, compute how far each column's statistics deviate from the historical mean. The most common measure is the z-score: (current_value - historical_mean) / historical_std_dev.
3. Flag significant deviations. If the z-score exceeds a threshold (typically 3), the metric is flagged as a behavioral anomaly. The severity depends on how extreme the deviation is: 3–4 sigma = low, 4–6 sigma = medium, 6+ sigma = high.
This is exactly how Sohovi's Behavioral Scoring works. After each completed run, the scorer compares the current run's column statistics against the last 10 runs and generates a Behavior Score (0–100) with per-column flags.
Types of Behavioral Anomalies
Null rate spike. A column that normally has near-zero nulls suddenly has a high null percentage. Almost always indicates an upstream data issue: a pipeline field not being populated, a join producing NULLs, or a source field being dropped.
Cardinality shift. A column that should be highly unique (like customer IDs) suddenly shows many duplicates. Or a low-cardinality categorical column suddenly has hundreds of unique values — suggesting a free-text field was accidentally substituted.
Sohovi automatically finds every duplicate in your dataset — including near-matches — and shows you exactly which rows are affected.
Mean drift. For numeric columns, the average value shifts dramatically. A revenue column averaging $200 that suddenly averages $20,000 is suspicious. Could indicate currency unit changes, data entry errors, or a filtering mistake.
Row count anomaly. The dataset normally has 50,000–55,000 rows but today has 8,000. A pipeline is broken, a filter is misconfigured, or the source data export was truncated.
Distribution shift. A categorical column that has always had a stable distribution of values suddenly has new values or a dramatically different frequency profile. Common when upstream systems add new codes or when a join goes wrong.
Type shift. A column that was consistently inferred as "date" now contains strings that don't parse as dates. The format may have changed upstream.
Why Behavioral Anomalies Are More Dangerous Than Rule Violations
Rule violations are visible. Your DQ score drops, flags appear, and you know something is wrong. You built the rule because you anticipated this failure mode.
Behavioral anomalies are often invisible — until the consequences appear downstream. A report shows the wrong numbers for two weeks before someone notices. A machine learning model starts making worse predictions. A customer receives communications based on stale or incorrect data.
The insidious nature of behavioral anomalies is that they look fine on the surface. The data loads. The columns are there. The formats are correct. Your static rules pass. But the statistical behavior has changed, and something downstream will eventually break.
When to Investigate a Behavioral Anomaly
Not every anomaly requires action. Some guidelines:
High-severity anomalies (6+ sigma) always warrant immediate investigation. A 6-sigma event has a probability of about 1 in a billion under normal distribution assumptions. If you're seeing one, something unusual happened.
Medium-severity anomalies (4–6 sigma) should be reviewed. These could be legitimate data events (a big sale day spiking revenue) or genuine problems. Check the upstream source.
Low-severity anomalies (3–4 sigma) are worth noting but may be noise. With enough metrics being monitored, some will fall in the 3-sigma range by chance.
When investigating, trace backward: check the source system, the extraction process, the transformation logic. The anomaly in your DQ run is a symptom — the root cause is upstream.
Key Takeaways
- Behavioral anomalies are deviations from historical patterns, not violations of fixed rules
- They're detected by comparing current statistics against a historical baseline using z-scores
- Common types: null rate spike, cardinality shift, mean drift, row count anomaly, distribution shift, type shift
- Behavioral anomalies are often silent — they pass static rules but indicate upstream problems
- High-severity anomalies always warrant investigation; low-severity ones may be noise
FAQ
Q: What's the difference between a behavioral anomaly and a rule violation? A: A rule violation is checked against a fixed threshold you defined. A behavioral anomaly is detected by comparing against historical patterns — no manual threshold required.
Q: How many historical runs are needed before anomaly detection is meaningful? A: At minimum 2, but 5–10 provides a more reliable baseline with lower false positive rates.
Q: Can behavioral anomaly detection generate false positives? A: Yes. Some metrics will fall in the 3-sigma range by chance. Review anomalies rather than automatically treating them as confirmed issues.
Q: How are z-scores interpreted? A: A z-score of 3 means the current value is 3 standard deviations from the historical mean. Higher z-scores indicate more unusual deviations.
Q: Do behavioral anomalies replace static rules? A: No — they complement them. Static rules enforce known constraints; behavioral anomalies catch unexpected deviations.
Q: What causes null rate spikes? A: Pipeline changes that stop populating a field, join conditions producing NULLs, source API changes removing a field, or ETL filter errors.
Q: What causes cardinality shifts? A: Duplicate introduction (joins gone wrong), free-text fields being mapped to what should be categorical columns, or IDs being re-used across different entities.
Q: What causes mean drift in numeric columns? A: Currency unit changes, data entry errors, filtering mistakes including or excluding different data segments, or legitimate business changes (e.g., price increases).
Q: How is Behavior Score calculated? A: Starting from 100, each high-severity flag subtracts 10 points, medium subtracts 5, low subtracts 2. Score cannot go below 0.
Q: Can I configure which columns are monitored for behavioral anomalies? A: Sohovi monitors all columns automatically. Enterprise tools like Collibra allow per-column configuration.
