You're deciding where in your data workflow to run quality checks, and you've heard the phrase "data quality at the source" — but you're not sure what that means in practice or whether it matters for a team your size.
Data quality at the source means validating or correcting data as close to its origin as possible — at entry, collection, or ingestion. Downstream quality checks happen after data has already moved into a pipeline, database, or reporting layer. The difference isn't just technical; it's a question of cost and leverage.
The Core Principle: Quality Problems Compound
Every step data travels without being checked is another step where it can be used incorrectly. A missing field that enters your CRM might generate an automated email with a blank name. That wrong email creates a customer service complaint. That complaint requires manual resolution. One quality problem at the source created a chain of downstream costs.
Sohovi profiles every column in your dataset for completeness and flags the exact rows where values are missing — free to try.
This compounding is why the data engineering community commonly describes fixing errors at the source as dramatically cheaper than fixing them in the middle or end of a pipeline. The specific ratios vary by organization, but the principle — that earlier is cheaper — is consistently supported in practice.
What "Quality at the Source" Looks Like in Practice
Source-level quality includes:
- Form validation — preventing invalid entries at point of data collection (required fields, format constraints, allowed values)
- API-level validation — rejecting records that don't conform to expected schema before they're written
- Pre-import checks — validating a file before loading it into a database or system
- Supplier or partner data contracts — agreeing on quality standards before receiving data from an external source
Sohovi lets you set up validation rules for any column and instantly see which rows fall outside them — no code or SQL required.
The key is that data is either corrected or rejected before it travels anywhere.
What Downstream Quality Checks Look Like
Downstream checks happen after data has been ingested. This includes:
- Quality monitoring in a data warehouse after ETL runs
- Profiling a dataset that's already been imported and is sitting in a database
- Running quality audits on existing records to remediate historical problems
- Detecting issues in reporting layers that trace back to upstream sources
Sohovi tracks quality trends across runs and alerts you when a metric — null rate, duplicate count, score — moves outside its normal range.
Downstream checks are valuable — particularly for detecting gradual degradation over time and for auditing data you've inherited. But they're reactive, not preventive.
The Real Cost Difference
Fixing a bad record at the source means the correction happens once, close to the person who has context to fix it correctly. Fixing it downstream means:
- Tracing the error back to its origin (time)
- Possibly updating multiple systems where the bad data has already propagated (more time)
- Dealing with any decisions that were made based on the bad data (cost)
- Notifying downstream stakeholders of the correction (overhead)
The earlier in the pipeline a problem is caught, the lower the correction cost.
When Downstream Checks Are Unavoidable
You can't always control data quality at the source. When you receive data from external partners, third-party systems, or historical exports, you don't control entry — you can only check what arrives. In these cases, downstream quality checks are the only option.
The practical approach: do everything you can at the source, and add downstream monitoring as a backstop for what you can't control.
Frequently Asked Questions
Q: What does "shifting left on data quality" mean? It's a borrowing from the software concept of "shift-left testing" — moving quality checks earlier in the lifecycle so problems are caught before they propagate. In data, it means moving quality validation as close to data entry or collection as possible, rather than downstream in the pipeline.
Q: Is source-level quality control always better than downstream? It's almost always cheaper to prevent than to remediate. But "better" depends on context. If you don't control the source — you receive a data file from a partner — downstream checks are necessary regardless of preference. And some quality problems (aggregation errors, cross-system consistency issues) can only be detected once data is assembled downstream.
Q: What role do data contracts play in source-level quality? A data contract is a formal agreement between a data producer and consumer defining the expected schema, format, and quality standards of data being exchanged. When enforced, it makes source-level quality a shared responsibility — the producer validates their output, not just the consumer.
Q: Can legacy systems support source-level quality controls? Often not without modification. Legacy systems with rigid input forms or batch export processes may not allow for quality rules to be injected at the point of entry. In these cases, the best available "source-level" option is validating data as it exits the legacy system — before it enters any downstream modern pipeline.
Q: What is "data quality at rest" versus "data quality in motion"? Data quality in motion refers to checking data as it's transferred or streamed through a pipeline. Data quality at rest refers to profiling and validating data that's already stored in a database or file. Both are downstream relative to the original source — but "in motion" catches problems sooner than "at rest."
Q: How do you enforce source-level data quality when the data comes from humans filling in forms? The most effective approaches are: required fields (the form won't submit without them), inline format validation (feedback appears as the user types), constrained input (dropdown menus instead of free text where possible), and confirmation steps for high-risk values (a confirmation dialog if an entered value seems unusual). Each constraint reduces the volume of bad data that needs downstream correction.
Q: What is "data quality by design"? It refers to building quality constraints into systems from the outset — defining validation rules during system design rather than retrofitting them after problems appear. It's the architecture-level version of shifting quality checks to the source.
Q: How do downstream quality checks interact with data lineage? Data lineage tracks where data came from and how it was transformed. When a downstream quality check flags a problem, lineage allows you to trace the error back to its source — identifying which upstream system or transformation introduced it. Without lineage, diagnosing the root cause of downstream quality issues is much harder.
Q: Should the team that owns the source data or the team that uses it downstream be responsible for data quality? This is a common organizational tension. The most effective model assigns primary responsibility to the source team (they own the data they produce) with downstream teams empowered to raise quality requirements back to the source. Data governance frameworks typically codify this through data stewardship roles.
Q: Is it ever too late to "shift left" on data quality? It's never too late to start moving quality checks earlier. Even if you're currently doing all quality work downstream, adding a pre-import validation step for new data is immediately achievable. Shifting left doesn't require re-architecting everything at once — it can be incremental.
Prevention at the source is cheaper than remediation downstream. The goal isn't to choose between the two — it's to push quality checks as far upstream as your architecture allows, while maintaining downstream monitoring as a backstop for what you can't control.
For teams receiving files from external sources — where source-level control isn't possible — Sohovi provides a fast, no-code quality gate: run a full check on the file before it enters any downstream system.