The performance of a machine learning model is bounded by the quality of its training data — a model trained on inaccurate, incomplete, or inconsistent data will learn to make inaccurate, incomplete, or inconsistent predictions.
The phrase "garbage in, garbage out" was coined for computing in general, but it applies nowhere more precisely than in machine learning. The difference is that in traditional software, garbage input causes obvious errors. In ML, garbage input causes confident-looking errors — the model produces predictions with high apparent certainty based on patterns learned from flawed data.
In this guide
- How each data quality dimension affects ML models differently
- The most common data quality problems that degrade model performance
- How to assess data quality before training
- What to check when a model isn't performing as expected
How Data Quality Dimensions Map to ML Failures
Accuracy — The Foundation of Model Reliability
A model learns patterns from its training data. If the labels or feature values in the training data are inaccurate, the model learns the wrong patterns.
Sohovi scores your dataset against your own accuracy standards and highlights the columns and rows where values fall outside expected ranges.
Consider a churn prediction model trained on customer data where a significant portion of "churned" labels are actually data entry errors — customers who were marked as churned because their records were deleted for a different reason. The model learns that certain characteristics predict churn when they actually predict data entry errors. The resulting model is not just wrong — it's confidently wrong.
Label accuracy is the most critical data quality dimension for supervised learning. Even small error rates in labels (5–10%) can meaningfully degrade model performance, particularly for imbalanced classification tasks.
Completeness — How Missing Data Distorts Training
Most ML frameworks handle missing data through imputation (filling in a value) or row exclusion (dropping rows with missing values). Both introduce bias:
Sohovi profiles every column in your dataset for completeness and flags the exact rows where values are missing — free to try.
Imputation bias: Replacing a null with the column mean assumes that missing values are distributed the same as present values. If they're not — if, for example, high-value customers are more likely to have missing phone numbers because they signed up through a different channel — the imputed values systematically misrepresent those customers.
Exclusion bias: Dropping rows with missing values removes a potentially non-random subset of the data. If the missingness is correlated with the target variable (customers who never provide an email are also more likely to churn), excluding them produces a model that was never trained on that population.
Consistency — Inconsistent Features Create Noise
ML models learn from feature values. When the same concept is stored inconsistently across records — country stored as "US", "USA", "United States", and "United States of America" — the model treats them as four separate features, diluting the signal and adding noise.
Sohovi profiles your training data for quality issues — missing values, outliers, type mismatches — before they corrupt your model.
For categorical features, inconsistency fragments the signal. Instead of learning that "United States" customers behave a certain way, the model learns four weak partial patterns that it may not generalize correctly.
Timeliness — Stale Data Trains On the Past
Models trained on outdated data learn patterns that may no longer apply. A model trained on 2019 purchasing behavior will have learned patterns that pre-date significant changes in consumer behavior. If the training data includes stale features (prices, product names, customer attributes that have since changed) alongside current labels, the model learns a corrupted mapping.
Uniqueness — Duplicates Over-weight Specific Records
Duplicate records in training data cause a model to see certain patterns more frequently than they actually occur, artificially increasing the weight of those patterns. If a particular customer type appears three times due to deduplication failures, the model treats that customer type as three times more common than it is.
For small training datasets, even a few dozen duplicates can meaningfully skew the learned distribution.
The Most Common Data Quality Problems That Degrade ML Models
Mislabeled training data
Systematic label errors are among the most damaging quality problems for supervised ML. Unlike random errors (which add noise but cancel out statistically), systematic mislabeling creates systematic biases that the model learns and replicates.
Common sources of systematic mislabeling: manual labeling processes with rater disagreements, rule-based labeling systems with flawed rules, label extraction from operational systems where the label definition changed over time.
Class imbalance amplified by missing data
When the minority class in an imbalanced dataset also has a higher rate of missing values, the effective imbalance is worse than it appears. The model sees even fewer high-quality examples of the minority class than the raw counts suggest.
Distribution shift between training and inference data
When the quality of data used in production (inference time) is different from the quality of training data, model performance degrades. A model trained on clean, well-formatted data will perform poorly when given messy production data that has nulls, inconsistent formatting, and dirty values that weren't present in training.
Feature leakage through data quality patches
When data quality problems are "patched" in training data (nulls imputed, duplicates removed, outliers capped) but those patches aren't applied consistently at inference time, the model encounters a different data distribution than it was trained on. This is a subtle but common source of training-serving skew.
Tools like Sohovi are particularly useful for analyzing the quality of datasets before they enter an ML pipeline — profiling CSV exports to surface null rates, outliers, and format inconsistencies that could introduce bias before training begins.
How to Assess Data Quality Before Training
Profile every feature: For each feature in your training dataset, measure: null rate, value distribution, outlier rate, and format consistency. Features with null rates above 20% require explicit handling strategy decisions (not default imputation).
Audit label quality: For supervised models, audit a random sample of labels manually. Even a 100-record audit can surface systematic label errors that would otherwise only be discovered after the model is deployed.
Check for data leakage features: Features that are correlated with the label for the wrong reasons — features that wouldn't be available at inference time, or features that were retrospectively added to the training data — will inflate training metrics but produce poor production performance.
Check for temporal consistency: Verify that the training data represents the time period you're predicting for. A model trained on historical data should reflect the same time period's conditions, not a mix of historical and current data.
Document the data lineage: Know where every feature came from and how it was processed before training. Undocumented transformations are a common source of feature drift between training and production.
What to Check When a Model Isn't Performing as Expected
When a deployed model's performance degrades or its predictions seem wrong:
Check for data distribution shift: Has the quality or distribution of production data changed since training? Run the same profiling checks on production data that you ran on training data and compare.
Check inference-time null rates: If production data has higher null rates for key features than training data, the model is operating outside its training distribution.
Check label quality if you have feedback: If you have ground truth labels for predictions (e.g., actual churn outcomes), audit the cases where the model was most confident but wrong.
Check for feature drift: Are the categorical values in production the same as those seen in training? New values that weren't in the training set are treated as missing by most frameworks.
Frequently Asked Questions
Q: How much does data quality affect ML model accuracy? Research consistently shows that data quality has a larger impact on model performance than algorithm choice for most real-world problems. A simple model trained on high-quality data often outperforms a sophisticated model trained on poor-quality data. Industry estimates suggest that data preparation and quality account for 60–80% of the work in a successful ML project.
Q: What percentage of mislabeled data is acceptable in a training set? It depends on the task and class balance. For balanced binary classification, random label noise up to 10–15% may have modest impact. For imbalanced problems where the minority class is the target (fraud detection, rare disease prediction), even 2–3% systematic mislabeling of the minority class can significantly degrade performance. There's no universal acceptable rate — measure impact empirically for your specific problem.
Q: Should I impute missing values or drop rows with missing data? Neither is universally correct. Dropping rows is defensible when missing data is rare and appears to be missing at random. Imputation is appropriate when missing data follows a systematic pattern but the mechanism is understood. In either case, document and justify the decision — and apply the same logic consistently at inference time.
Q: How do duplicates in training data affect model performance? Duplicates artificially over-represent specific examples in the training distribution. The model effectively sees those examples multiple times, increasing the weight of their associated patterns. For large datasets with few duplicates, the impact is minimal. For small datasets where duplicates represent a significant percentage of records, the impact on learned distributions can be substantial.
Q: What is training-serving skew and how does data quality cause it? Training-serving skew occurs when the data distribution at inference time differs from the training distribution. Data quality is a common cause: training data was cleaned and preprocessed in ways not replicated in production, or production data quality has drifted since training. The model encounters patterns at inference time that don't match its learned representations.
Q: How do you detect data quality problems in a training dataset? Profiling: compute descriptive statistics (mean, median, mode, null rate, unique count, value distribution) for every feature. Visualization: plot feature distributions and look for unexpected patterns, gaps, or outliers. Cross-validation: if performance varies significantly across folds, this may indicate inconsistent data quality across different record subsets.
Q: Does more training data compensate for lower data quality? For random noise, larger datasets do help because statistical errors average out. For systematic quality problems (systematic mislabeling, consistent feature corruption, feature leakage), more data amplifies the problem rather than mitigating it. More examples of a flawed pattern teaches the model the wrong thing more confidently.
Q: What's feature leakage and how does it relate to data quality? Feature leakage occurs when a feature is correlated with the label for a reason that won't hold at inference time. In the context of data quality, this often happens when data quality patches or post-processing steps applied to training data "know" the outcome in ways that production data won't. For example, filling in missing values using information that was only available after the event being predicted.
Q: How should data quality checks be integrated into an ML pipeline? Add automated quality checks at the data ingestion step of the training pipeline: check null rates, value distributions, duplicate rates, and schema consistency against defined thresholds. Fail the pipeline if quality checks don't pass — don't train on bad data automatically. Apply the same checks to inference-time data and alert when production data drifts outside training-time thresholds.
Q: Is data quality more important for deep learning than for traditional ML? Deep learning models are generally more sensitive to data quality because they learn more complex patterns and are more capable of overfitting to noise in the training data. They also require larger datasets, which means data quality problems are present at larger scale. Traditional models with explicit feature engineering are more resistant to some data quality problems because domain knowledge is built into the features rather than learned.
If you want to assess the quality of a dataset before using it for ML training, Sohovi can profile any CSV in minutes — showing null rates, value distributions, outliers, and format issues that could bias your model. Try it free — no code, no setup required.
