Data distribution shift happens when the frequency profile of values in a column changes between data loads — and it's nearly invisible to static rule-based quality checks.
Your static rules check that values are valid, non-null, and correctly formatted. But they don't check whether the mix of values has changed. If your "payment_method" column used to be 70% credit card and 30% bank transfer, and now it's 5% credit card and 95% bank transfer — all your rules pass, but something has clearly changed upstream.
That's a distribution shift. And it's one of the sneakiest data quality problems in existence.
Why Distribution Shift Is Hard to Catch
Static rules are well-suited for absolute constraints: a field must not be null, a value must be within a range, a format must match a pattern. These checks don't depend on history — they evaluate each value in isolation.
Distribution shift requires historical context. To know that the proportion of "credit card" payments is anomalously low, you need to know what "normal" looks like. That means storing historical value frequency data and comparing the current run against it.
Sohovi profiles your datasets for quality issues in minutes — see what's broken before it breaks your pipeline — try Sohovi free.
Most lightweight DQ tools don't do this. Enum validation rules can check that a value is one of the allowed set, but they don't check whether the proportion of each value has shifted.
Types of Distribution Shift
New value emergence — A value appears in the current run that was never present in historical data. Example: "CANCELLED_FRAUD" appearing as a payment status when only "COMPLETED", "REFUNDED", and "FAILED" existed before.
Dominant value shift — The top value changes between runs. Example: "active" was 65% of the "status" column; now it's 8%. "churned" went from 5% to 72%.
Frequency collapse — A value that was previously common effectively disappears. Example: "manual" order type drops from 40% to 0.1%.
Frequency explosion — A value that was previously rare becomes dominant. Example: a product category "other" goes from 2% to 89%.
How Automatic Detection Works
The most common approach is to store the top N values and their frequencies after each run, then compare the current run's distribution against the previous run (or against a rolling average of recent runs).
Specific detection patterns:
New value detection: If value V appears in the current run's top values but was absent from the previous run's top values, flag it — especially if V represents a significant percentage (e.g., >5% of rows).
Top value change detection: If the most frequent value changes from run to run, flag it. This is a strong signal of a significant distributional shift.
Shannon entropy comparison: Compute the entropy of the value distribution for each run. A large change in entropy indicates a significant shift in how values are distributed, even if no individual value is new.
Sohovi's behavioral scoring uses the first two approaches: it flags new values in the current run that were absent from the previous run, and it specifically flags when a new value becomes the dominant entry (>10% frequency) where one wasn't before.
Real-World Examples
E-commerce returns: Your "return_reason" column historically shows "damaged", "wrong_item", "changed_mind". A new run shows "qa_defect" at 34% frequency — a product quality issue that no static rule would catch.
CRM sync issues: Your "lead_source" column was 50% "organic", 30% "paid", 20% "referral". A pipeline bug starts marking everything as "import" — distribution shifts to 95% "import" — real lead source data is lost.
Sohovi finds gaps, duplicates, and format errors in your CRM data — so your team is working from records they can trust.
Status code changes: An upstream API updates its status codes from numeric (1, 2, 3) to string ("active", "inactive", "pending"). Distribution shift detection flags that the numeric values disappeared and new string values appeared.
Data refresh failures: A partial pipeline run produces a batch where only a subset of regions is included. The "region" column's distribution shifts dramatically — "Northeast" drops from 25% to 2% because only that region's data was refreshed.
Implementing Detection Without SQL
You don't need SQL to implement distribution shift detection. The algorithm is straightforward:
- After each run, store the top 10–20 values and their frequencies for each column.
- When a new run completes, compare its top values against the previous run's top values.
- Flag any value in the new run's top values that was absent from the previous run's top values.
- Flag if the top-1 value changed.
Sohovi implements this automatically — the profiler stores top values for every column, and the behavioral scorer compares them between runs. Distribution shift flags appear in the Behavior Score card on every run detail page, with no configuration required.
How to Respond to a Distribution Shift Alert
When you see a distribution shift flag:
- Confirm it's real — check the raw data or upstream source to verify the distribution actually changed.
- Classify it — is this an expected change (new product line, new customer segment) or a data quality issue?
- Trace the source — if unexpected, trace backward through your pipeline to find where the change was introduced.
- Update baselines if legitimate — if the shift is real and expected, the baseline will naturally update over subsequent runs.
Key Takeaways
- Distribution shift is a change in the frequency profile of categorical column values
- Static rules cannot detect it because they don't compare against historical distributions
- Automatic detection requires storing historical value frequencies and comparing across runs
- Common signals: new values appearing, dominant value changing, frequency collapse or explosion
- Sohovi's behavioral scoring flags distribution shifts automatically with no configuration
FAQ
Q: Do I need to configure which columns to monitor for distribution shift? A: No. Sohovi monitors all columns automatically.
Q: What percentage threshold triggers a distribution shift flag? A: A new value is flagged if it appears in the current run's top values but was absent from the previous run. A "dominant value shift" is flagged if a new value reaches >10% frequency.
Q: Does distribution shift detection work for numeric columns? A: For numeric columns, behavioral scoring uses statistical checks (mean, std dev, min/max) rather than value-frequency comparison. Distribution shift detection specifically targets categorical/string columns.
Q: How do I know if a distribution shift is a bug or a legitimate business change? A: Check the upstream source. If the shift is reflected in the source system intentionally, it's likely legitimate. If the source hasn't changed but the extracted data has, it's likely a pipeline bug.
Q: Can I suppress distribution shift alerts for expected changes? A: This is a roadmap feature. Currently, you can acknowledge the flag on the run detail page.
Q: How many historical runs does distribution shift detection need? A: It compares against the immediately previous run — so you need at least 2 runs for any detection to fire.
Q: What if a column legitimately gains new values over time? A: Over subsequent runs, the new value will appear in the historical distribution and will no longer be flagged.
Q: Is this the same as concept drift? A: Related but different. Concept drift is an ML term about model performance. Distribution shift in DQ refers to changes in the data being processed, not a model's predictions.
Q: Does this work for high-cardinality columns? A: For high-cardinality columns (e.g., customer IDs), value-frequency detection is less meaningful. Sohovi uses statistical checks (null rate, unique rate) for those instead.
Q: Can I export a report showing distribution shifts over time? A: This is a roadmap feature. Currently, distribution shift flags are visible per-run on the run detail page.
