Reviewing Survey Responses

As responses begin to come in, it's best practice to regularly download raw data and review responses for quality. Ensuring clean, trustworthy data is critical before beginning any analysis.

Here are a few things to watch for:

• Check the length of time taken to complete the survey.

• Flag responses that are completed significantly faster than average (e.g. under 1/3 of median time).

• Some speeders may be careless or not reading questions fully.

Pro Tip: Create a time-based threshold (e.g., “under 2 minutes” for a 10-minute survey) and review manually before deletion.

• Also look for respondents who take much longer than expected, which can sometimes indicate distraction or technical issues.

• These may not always need to be removed but should be noted.

• If your survey includes a quality check question (e.g. “Please select option C for this question”), use it to flag inattentive respondents.

• Anyone who fails this should be considered for removal.

Pro Tip: Don’t rely on just one check—combine with speed or text review.

• Review open-end responses for signs of:

  • Nonsense or irrelevant answers (e.g. “asdf,” “yes” to a “why” question)

  • Copy/paste or repeated robotic phrases

  • Identical responses across multiple questions

Pro Tip: Sort open ends by length or use keyword search to spot junk responses quickly.

• If a respondent gives the same answer across a long battery of scale questions, it may indicate disengagement.

• This is especially important when testing concepts or comparing attributes.

• Depending on your survey logic, incomplete surveys may still be saved.

• Review for skipped key questions that would impact your analysis.

• Ideally: Review data after the first 50–100 completes, then again at 25%, 50%, and final fielding.

• Always do a final clean before exporting reports or running crosstabs.
  

Once you've completed your data quality review and identified which responses should be excluded, you can remove them directly within the platform.

1. Navigate to the Manage Responses page
   Go to the Analyze Results tab and click Manage Responses from the left-hand sidebar.

2. Identify which box to use
   You'll see two separate removal options:

   • Remove for Quality Issues
     Use this box for respondents flagged due to:

     • Speeding through the survey

     • Failing red herring checks

     • Robotic or copy/paste open-ends

     • Straightlining behavior

   • Remove for Non-Quality Reasons
     Use this if you need to rebalance your sample (e.g., too many from a particular region or demographic group) even though their responses were valid.

3. Paste in the Respondent IDs
   Copy and paste one or more Respondent IDs into the appropriate box. Each ID should be on its own line or separated by commas.

4. Click “Remove Respondents”
   After pasting the IDs, click the Remove Respondents button to confirm.

5. Wait for the page to refresh
   The platform will reprocess your data. Wait a few moments while the removal is completed.

6. Return to the Analytics tab
   Navigate back to the Survey Summary on the Analytics tab to confirm your new total respondent count reflects the updated dataset.
   

Pro Tip: Keep a log of which respondent IDs you removed, and why (e.g. speeders, failed attention checks, sample balancing). This helps ensure transparency and consistency in future reports.