Adaptive data quality is a monitoring approach that learns what "normal" looks like for your data and automatically flags deviations — without you having to write a rule for every possible failure mode.
Traditional data quality tools work by checking whether data satisfies a set of hand-crafted rules: is the email formatted correctly? Is the value within a specified range? Is the required field populated? These rules work well for problems you know about in advance. But data breaks in ways you never anticipate — and static rules are silent when that happens.
Adaptive data quality solves this by building a statistical baseline from your historical data and raising alerts whenever a new batch deviates significantly from that baseline.
Sohovi tracks quality trends across runs and alerts you when a metric — null rate, duplicate count, score — moves outside its normal range.
Why Static Rules Fail in Practice
Static rules have three fundamental limitations.
They only catch what you predicted. If your data engineer wrote a rule that checks whether null rate is below 10%, a jump from 0.5% to 8% slips through with no warning. A jump from 0.5% to 80% fails the rule — but by then, the damage may already be done downstream.
They don't account for context. A 5% null rate might be completely normal for your "notes" column but catastrophic for your "customer_id" column. Static thresholds treat all columns equally unless you manually tune each one — which takes hours and goes stale as data evolves.
They can't detect unknown unknowns. A new upstream system integration might start sending data in a slightly different format. A pipeline bug might start duplicating a subset of rows. A new value might appear in a categorical field that downstream code doesn't handle. None of these have a pre-written rule.
How Adaptive Data Quality Works
The core mechanism is statistical: after each run, the system stores summary statistics for each column (null rate, cardinality, mean, standard deviation, min, max, dominant values). Over time, this builds a baseline of what "normal" looks like.
When a new batch arrives, each column's statistics are compared against the baseline. The comparison uses z-scores: how many standard deviations away is the new observation from the historical mean? If the deviation exceeds a threshold (commonly 3 sigma), the system flags it as a behavioral anomaly.
This means the system automatically learns:
- Your "customer_email" column normally has 0.5% nulls — a run with 15% nulls is a 6-sigma event
- Your "order_value" column averages $127 with a std dev of $40 — a run averaging $890 is suspicious
- Your "status" column has always been one of four values — a new value "SUSPENDED_FRAUD" appearing at 22% is a distribution shift
None of these require a hand-crafted rule. The system catches them automatically.
Sohovi's Behavioral Scoring implements exactly this approach — after two completed runs on an asset, every subsequent run generates a Behavior Score (0–100) alongside the traditional DQ Score, with per-column flags showing exactly which metrics shifted and by how much.
Adaptive Quality vs Static Rules — Which to Use?
You need both. Static rules encode known business requirements: an email field must match the email regex, an age must be positive, a required field must be populated. These are non-negotiable constraints that adaptive learning can't replace.
Adaptive scoring catches everything else — the unexpected, the gradual drift, the pipeline bug you didn't know happened. Think of static rules as your contract with the data and adaptive scoring as your canary in the coal mine.
The Metrics That Matter Most
The most valuable metrics to monitor adaptively are:
- Null rate per column — the most common signal of upstream data issues
- Cardinality — if a supposedly-unique column starts having duplicates, or a low-cardinality enum suddenly has hundreds of values
- Mean and standard deviation — for numeric columns, sudden shifts in average value indicate a data problem
- Row count — a pipeline that normally produces 50k rows and suddenly produces 500 rows has probably broken
- Dominant values — for categorical columns, new values that weren't in the historical distribution
When Behavioral Scoring Beats Static Rules
Consider a monthly customer file that normally contains 45,000–52,000 rows. One month, the pipeline produces 12,000 rows because a filter condition was accidentally changed. Your static rules might all pass — the nulls are fine, the formats are fine — but the row count is 75% below normal. Only adaptive scoring catches this.
Or consider a status column that historically contains "active", "inactive", "pending". A new upstream system starts sending "ACTV", "INACTV", "PEND" — your enum validation rule fails, but it's unclear why. Adaptive distribution shift detection tells you immediately: three new values appeared that weren't in the previous run's distribution.
Key Takeaways
- Static rules catch known problems; adaptive scoring catches unknown deviations
- Behavioral baselines are built automatically from run history — no manual configuration needed
- Z-score comparison flags statistically unusual changes even when static rules pass
- The combination of static rules + adaptive scoring gives you full coverage
FAQ
Q: How many historical runs do I need before adaptive scoring works? A: Most implementations (including Sohovi) require at least 2 historical completed runs to establish a baseline. The more runs you have, the more accurate the baseline.
Q: Can adaptive scoring replace static rules entirely? A: No. Static rules encode business requirements that adaptive learning can't infer. Use both together.
Q: What is a z-score in this context? A: A z-score measures how many standard deviations an observation is from the historical mean. A z-score of 3 means the value is 3 standard deviations away — which is statistically unusual enough to flag.
Q: What is a "normal" z-score threshold for flagging? A: Most implementations use 3 as the minimum threshold. Values with z-scores of 4–6 are medium severity; above 6 is high severity.
Q: Does adaptive scoring work for small datasets? A: It works, but it's more meaningful with larger datasets and more run history. For small datasets, the baseline is more volatile.
Q: What's the difference between adaptive scoring and anomaly detection? A: They're closely related. Adaptive scoring is a form of time-series anomaly detection applied to data quality metrics.
Q: How does distribution shift detection work? A: It compares the most frequent values in a column between the current run and the previous run. New dominant values that weren't present before are flagged.
Q: Can I configure the z-score threshold? A: In most systems, yes. Sohovi's behavioral scorer uses a threshold of 3 by default (flagging at 3+ sigma deviations).
Q: Is my data sent to a server for behavioral scoring? A: Sohovi computes behavioral scoring from aggregated statistics (null rates, averages, value counts) — never from raw row data. Your data stays in your browser.
Q: How is the Behavior Score calculated? A: Each behavioral flag subtracts points based on severity: high severity flags subtract 10 points, medium subtract 5, low subtract 2. The score starts at 100 and cannot go below 0.
