Survey data starts with a structural quality problem: respondents control the inputs. Unlike a CRM where you control what fields exist and can enforce validation, surveys depend on honest, thoughtful responses from people who may be rushing, not paying attention, or actively trying to provide bad data.
Types of Survey Data Quality Problems
Straight-lining: A respondent selects the same response for every question in a grid — all 4s, or all "Strongly Agree" — without reading the questions. This is a strong indicator of low-quality, inattentive response.
Contradictory answers: A respondent reports being 25 years old on one question and 45 years old on a follow-up. Or reports being a manager who reports to no one and has no direct reports. Internal contradictions flag responses that need review.
Speed violations: Respondents who complete a 10-minute survey in 90 seconds are almost certainly not reading the questions. Completion time is a useful quality filter.
Implausible demographics: Ages, incomes, or other demographic data that falls outside plausible ranges or contradicts stated qualifications.
Sohovi profiles your datasets for quality issues in minutes — see what's broken before it breaks your pipeline — try Sohovi free.
Open-ended gibberish: Free-text responses that are random characters, repeated text, or obvious non-answers ("asdfghjkl," "I don't care," "123456").
Duplicate responses: The same respondent completing the survey multiple times — identifiable through IP address, device fingerprint, or identical or near-identical response patterns.
Quality Filters to Apply Before Analysis
Time-based filtering: Flag responses completed in less than X seconds (typically one-third of the median completion time). Review or remove these.
Straight-line detection: Calculate the variance in responses across scales and grids. Responses with zero or near-zero variance are candidates for removal.
Consistency checks: For any question pair that has a logical relationship, flag contradictions. Age vs. years of experience. Role vs. company size. Education vs. age.
IP-based duplicate detection: Flag multiple submissions from the same IP address within a short time window.
Sohovi automatically finds every duplicate in your dataset — including near-matches — and shows you exactly which rows are affected.
Open-text quality check: Minimal length, random character detection, or key-phrase filtering for free-text fields.
Frequently Asked Questions
Q: What is survey data quality and why does it matter? Survey data quality refers to whether responses accurately represent respondents' actual opinions, behaviors, and characteristics. Low-quality responses — from inattentive or dishonest respondents — corrupt aggregate statistics and can lead research to wrong conclusions.
Q: What is straight-lining in surveys? Straight-lining is when a respondent selects the same answer for all items in a scale or grid without differentiating between questions. It typically indicates the respondent isn't reading the questions carefully. It can be detected statistically by looking for responses with zero variance across scale items.
Q: How do I detect speeders in survey data? Track completion time for each response. Flag responses completed in less than one-third of the median completion time (or a predetermined minimum based on reading speed). Review flagged responses for other quality indicators before deciding whether to remove them.
Q: What percentage of survey responses are typically low quality? It varies significantly by survey mode, incentive structure, and panel quality. For online panels, research suggests 5-30% of responses may exhibit at least one quality concern. For opt-in surveys of your own customers, quality is typically higher.
Q: Should I remove all low-quality survey responses or just flag them? A conservative approach: remove responses with multiple severe quality indicators (speeders + straight-liners + impossible demographics). Flag single-indicator responses for sensitivity analysis — run your analysis with and without them to see how much they affect the results.
Q: What are attention check questions? Attention check questions are deliberately simple or obvious questions embedded in a survey to verify that respondents are reading. "What is 2+2?" or "Please select 'Strongly Agree' for this question." Respondents who fail attention checks are likely to have lower quality on other items.
Q: How do I handle contradictory responses in survey data? Flag them as potential quality issues. For some contradictions, the explanation is legitimate (someone who is 25 and has 10 years of experience may have started work at 15). For others (a current employee who says they have no employer), the contradiction is implausible. Investigate the most severe contradictions before including those responses.
Q: What is satisficing in survey research? Satisficing is when respondents use cognitive shortcuts to answer questions — not because they're dishonest, but because answering every question thoughtfully is cognitively demanding. Straight-lining is one form of satisficing. Other forms include selecting the first acceptable answer rather than the best answer, or avoiding extreme response options.
Q: How does incentive structure affect survey data quality? High incentives relative to survey length attract respondents who complete surveys for compensation rather than genuine participation, increasing the likelihood of low-quality responses. Balancing incentives with qualification screening maintains response quality.
Q: What is the role of quota management in survey data quality? Quota management ensures that your sample matches target demographic characteristics (age, gender, industry). Without quotas, your sample may over-represent certain groups, producing biased results even if individual responses are high quality.
Survey data quality is a pre-analysis necessity, not an afterthought. Apply quality filters before running any analysis, document which responses were removed and why, and report the effective sample size after filtering.
