From Questionnaire to Live Survey: A Five-Phase Survey Testing Checklist

A survey testing checklist is the cheapest insurance a research team can buy. Once a link goes to thousands of panellists, every routing error, broken pipe or missing quota becomes expensive to fix and impossible to undo for the people who have already answered. This checklist follows a questionnaire from the final approved document to a stable, fully launched survey, phase by phase.

Why a written checklist beats "having a look"

Most teams test surveys. Fewer test them the same way every time. When testing relies on whoever happens to click through the link, coverage depends on their mood, their deadline and how well they know the questionnaire. A written checklist turns testing into a repeatable process: the same checks, in the same order, signed off by named people.

It also helps with accountability. Professional codes expect researchers to take care with participants and with the quality of what they deliver; the MRS Code of Conduct, for example, sets out professional standards for research practitioners, including attention to participant wellbeing. A disciplined survey QA process is one practical way of living up to those expectations.

Phase 1: Before programming starts

Testing begins before a single question is built. Many errors found in QA trace back to an unclear questionnaire, so the first checks are on the document itself.

• Questionnaire is the final approved version, with a version number and date
• Every question has a clear base (who is asked)
• Routing instructions reference specific question numbers, not descriptions
• Answer codes are defined, including "Other", "None" and "Don't know"
• Randomisation and anchoring instructions are explicit
• Quota definitions match the sample plan
• Termination and over-quota rules are written down
• Required languages and countries are confirmed

If any item is missing, raise it with the questionnaire author now. A query answered on day one costs minutes; the same query discovered during soft launch can cost a day of fieldwork.

Phase 2: Programmer self-check

The person who built the script should run through it before anyone else sees it. This is not the main QA, but it removes the obvious problems so that reviewers can focus on subtler ones.

• Walk the main qualifying path from start to complete.
• Force each screen-out and confirm the respondent exits with the right status.
• Check every piped value shows correctly, including when the source question offers "Other (please specify)".
• Confirm randomised lists actually randomise, and anchored options stay put.
• Check validation messages are clear and appear in the correct language.
• Review the datamap: every question has a variable, every variable has labels.

The logic itself, including skip, display and piping rules, is covered in more depth in our survey programming logic guide.

Phase 3: Independent survey QA

Independent testing is the heart of the process. The tester should not be the programmer, and ideally should have the questionnaire open side by side with the test link.

Content checks

• Question wording matches the approved questionnaire exactly.
• Answer lists are complete, correctly ordered and correctly coded.
• Instructions ("select all that apply") match the question type.
• Spelling, grammar and brand names are correct in every language version.

Logic checks

• Each routing instruction is tested in both directions (the condition met and not met).
• Display logic at option level is tested with several combinations of earlier answers.
• Loops repeat the right number of times with the right piped item.
• Derived variables calculate correctly; check them in the test data, not just on screen.

Device and experience checks

• The survey works on a recent mobile browser and on desktop.
• Grids are usable on a small screen, or switch to a mobile-friendly layout.
• Progress indicators behave sensibly and the survey length feels reasonable.
• Images, videos or concept boards load and display at a readable size.

Who checks what

Phase 4: Soft launch

A soft launch sends the live survey to a small share of the sample before full release. Its purpose is to catch what testing cannot: real respondents behaving in unexpected ways, incidence that differs from the estimate, and technical issues that only appear at volume. The AAPOR best practices for survey research recommend pretesting the questionnaire and piloting procedures before fielding, and a soft launch is the practical version of that for most online studies.

1. Release a small batchSend a limited number of invitations across key sources and countries.
2. Pause and review dataCheck completes, drop-off points and screen-out reasons.
3. Compare against assumptionsLook at incidence and length of interview versus the costing.
4. Review open-ends and quality flagsScan for confusion, speeders and straight-lining.
5. Agree changes in writingDocument any fixes and retest affected paths before resuming.

What to look for in soft-launch data

• Drop-off concentration: if many respondents abandon at one question, it may be confusing, too long or broken on mobile.
• Unexpected screen-outs: a screener that terminates far more people than expected may have a logic error rather than a low incidence.
• Length of interview: if the median is well above the estimate, raise it with the client and the sample supplier before costs drift.
• Open-ended answers: confused or irritated comments are often the fastest signal that something is wrong.
• Quota cells: confirm completes are counting towards the right cells.

Data quality checks during and after soft launch are covered further in our guide to online survey data quality checks.

Phase 5: Full launch and the first day in field

Going live is not the end of testing. The first hours of full fieldwork are when volume exposes problems that a small batch did not.

• All soft-launch fixes are applied and retested
• Changes are logged with version number and approver
• Quotas are open, with targets matching the agreed plan
• Supplier redirect links for complete, screen-out and over-quota are confirmed
• Field monitoring is set up for completes, incidence and drop-off
• A named owner is watching the survey during the first hours
• Client has approved launch in writing

Once the survey is live, the work shifts to monitoring. Our article on live fieldwork monitoring covers what to watch day by day, from quota fill to supplier performance.

Keep a test log, not just a test link

A checklist tells you what to test; a test log records what was found and what happened next. Without one, the same issue gets reported three times by three people, or a fix is assumed to be done because someone mentioned it in a chat message.

A simple log needs only a few columns:

• Issue reference and date, so items can be discussed without confusion.
• Question number and path, describing how to reproduce the problem.
• Description and severity, separating blocking errors from cosmetic ones.
• Owner and status, so it is clear who is fixing what.
• Retest result, confirmed by the person who raised the issue rather than the person who fixed it.

Severity matters because time before launch is limited. A routing error that sends qualified respondents to a screen-out must be fixed before launch; a slightly awkward line break on one tablet layout may not. Agree with the project manager which severities block launch, and record that decision in the log.

Keep the log with the project once fieldwork starts. If the client later asks why a question changed after soft launch, the answer, along with who approved it, should be easy to find.

Common mistakes the checklist prevents

Most launch problems fall into a small number of patterns:

1. Testing only the "happy path": the qualifying route works, but nobody forced the over-quota exit.
2. Retesting only the changed question: a fix at Q8 breaks a condition at Q30 that referenced it.
3. Skipping mobile: a large grid becomes unusable on a phone and drop-off climbs.
4. Not checking redirects: completes are recorded in the survey but not passed back to the sample supplier, which creates reconciliation headaches later.
5. Undocumented changes: a quick edit during fieldwork means the data no longer matches the questionnaire on file.

Each of these is avoidable with a checklist and a named sign-off at each phase.

Where Sampledge fits

SampCode is the survey programming product in the Sampledge ecosystem, built by Frequent Research for teams that script and launch surveys routinely. Alongside SampTrek, survey work can sit within the same project record as quotas, suppliers and fieldwork tracking, so the people running QA, soft launch and live monitoring are looking at the same project rather than separate spreadsheets.

To see how this would work with your own launch process, talk to our team or compare workspace options on the pricing page.