Online Survey Data Quality Checks: A Layered Approach to Speeders, Bots and Duplicates

Online survey data quality is decided long before anyone opens the data file. It depends on how the questionnaire is designed, which checks are scripted into it, how fieldwork is watched and how cases are reviewed before delivery. This guide sets out a layered approach to speeder detection, straight-lining, duplicate respondents, bot detection and attention checks, and explains why no single check should be trusted on its own.

Why single checks are not enough

Many teams rely on one or two rules, typically a minimum completion time and a trap question. These help, but the evidence suggests they miss a lot. In a study of opt-in online polls, Pew Research Center found that among the respondents it classed as bogus, 87% passed a speeding check and 84% passed an attention check. The report, Assessing the Risks to Online Polls From Bogus Respondents, also found that bogus cases tended to give positive answers rather than random ones, which can quietly push results in one direction.

The practical lesson is that quality comes from layers: design that discourages poor answers, in-survey checks that flag them, and review that weighs several signals together.

Layer 1: design that discourages poor data

The cheapest quality control is a questionnaire people can complete honestly without getting bored or confused.

• Keep it as short as the objectives allow. Every extra question has a cost in attention, especially near the end.
• Break up long grids. Large batteries of statements are where straight-lining usually starts. Split them, or present items one at a time on mobile.
• Make questions answerable. Respondents who cannot find their answer pick something anyway. Include sensible "none", "don't know" or "other" options.
• Design for mobile first. Many respondents will take the survey on a phone, and layouts that work on a desktop can be painful on a small screen.
• Write a clear introduction. People who understand the purpose and expected length are more likely to stay engaged.

Getting these details into the script correctly is part of the programming job. Our survey programming logic guide and questionnaire to live survey checklist cover the build in more detail.

Layer 2: checks built into the survey

These are flags captured during the interview. They should be agreed with the client and written into the questionnaire before scripting starts.

Speeder detection

Record the time taken for the whole interview and, ideally, for key sections. Rather than an arbitrary fixed limit, base the threshold on the observed median length in a soft launch, and flag interviews completed in a small fraction of it. Section-level timing helps catch people who rush one long grid but take a normal time elsewhere.

Straight-lining

Straight-lining means giving the same answer to every item in a grid. Some of this is legitimate: a respondent can genuinely agree with every statement. The flag becomes meaningful when the grid includes items worded in opposite directions, so identical answers are logically inconsistent.

Attention checks

Instructed-response items ("please select 'somewhat disagree' for this row") and simple factual questions show whether someone is reading. Use them sparingly, keep them polite and avoid trick wording that a careful respondent could reasonably misread. As the Pew findings above show, passing an attention check is not proof of a genuine respondent.

Consistency checks

Ask related questions in different places and compare. A respondent who says they have never used a category and later rates three brands in it on usage frequency is worth reviewing. Age, role and company size in B2B screeners are common places to test consistency.

Open-ended review

Open text often reveals the clearest signs of low-quality data: gibberish, copied text, answers unrelated to the question, or the same generic phrase given to every open-end. Pew's report noted that some bogus open-ends read like generic product feedback regardless of what was asked. A short open question early in the survey makes a useful screen.

Layer 3: identity and technical signals

Some problems cannot be seen in the answers, only in how the interview arrived.

Duplicate respondents are a particular risk when several sample suppliers are used on one project, because the same person may belong to more than one panel. De-duplication should happen across suppliers, not only within each one.

Bot detection keeps changing, and no list of techniques stays complete for long. The principle is stable: combine technical signals with answer-based ones, and watch for sudden changes in the pattern of incoming completes.

Turning flags into decisions

A flag is a reason to look, not an automatic verdict. Removing every case that trips one rule throws away genuine respondents and can bias the sample towards people who answer in a particular way.

Setting thresholds sensibly

Thresholds should reflect the survey in front of you, not a number carried over from the last project. A few principles help:

• Base timing rules on real data. Use the soft launch to see how long genuine respondents take, then set the speeder threshold relative to that.
• Weight flags by strength. A completed hidden field or a duplicate ID is strong evidence on its own. A single fast section or one straight-lined grid is weak evidence that needs support from other flags.
• Allow for the audience. Senior B2B respondents and healthcare professionals often answer quickly because they know the subject well. Consumer studies on unfamiliar topics may show longer, more variable timings.
• Check the effect on quotas. If removals fall heavily on one age group, region or supplier, look again. That pattern may reveal a sourcing problem, or a rule that is penalising a particular kind of respondent.

Write the final thresholds into the rulebook and do not change them quietly mid-field. If a change is genuinely needed, record it with a date and apply it consistently to all cases.

A workable cleaning routine looks like this:

1. Agree the rulebookDefine each check, its threshold and how many flags lead to removal, before launch.
2. Review the soft launchCheck timings, drop-off and open-ends on the first completes, and adjust the rules.
3. Monitor daily in fieldWatch flag rates by supplier and by quota cell so problems are caught early.
4. Score each caseCombine flags into a simple score rather than acting on any single one.
5. Review borderline casesA person reads the open-ends and answer patterns before deciding.
6. Document and reconcileRecord removals with reasons and share them with suppliers.

Daily monitoring matters because a problem found on day one costs a few cases, while the same problem found at the end can cost the quota. Our guide to live fieldwork monitoring covers what to watch while a project is in field.

Quality and reconciliation go together

Every removed case has a commercial consequence. Suppliers are usually paid for accepted completes, so removals need to be documented in a way that holds up when invoices are agreed. Keeping respondent IDs, flag results and reasons together makes that conversation much simpler. We cover the process in sample supplier reconciliation.

• Each removed case has a respondent ID, supplier and reason
• Rules match the ones agreed before launch
• Replacement completes are tracked against the right quota cell
• Removal lists are shared with suppliers before invoicing
• The final dataset and the removal log are archived together

Where Sampledge fits

Sampledge supports quality at two points in this process. SampCode is where the questionnaire is scripted, including the timing, routing and question logic that in-survey checks depend on. SampTrek keeps the project record that quality decisions feed into: suppliers, quotas, live fieldwork progress and reconciliation, so removals and replacements stay tied to the right supplier and cell. If you buy online sample directly, SampAudience lets you track a launched study as it fills.

For the wider context, see our overview of data collection methods in research and our guide on how to choose a data gathering method. To see the workflow end to end, request a demo.