Set your target
Real Startup Book

Open-Ended Survey

Fixed set of questions, usually via a mass-communication channel such as an email or website pop-up.

Set your target
 
Blog Posts & Articles
Validation
Content

Open-Ended Survey

In Brief

An open-ended survey asks a fixed set of questions, usually via a mass-communication channel such as an email or website pop-up. Answers are not constrained as in the case of multiple choice or check boxes, but are free-text responses in which the customer can choose the length and detail of their answer.

Helps Answer

  • Who is our customer?
  • What are their pains?
  • What are the jobs to be done?

Tags

  • B2C
  • B2B
  • Qualitative
  • Customer
  • Generative market research

Description

Time Commitment and Resources

Surveys can be quick to write and execute, often taking only 1-2 hours to set up. However, designing effective questions that don’t introduce biases into customers' answers can require a high degree of skill, multiple revisions, and even comprehension tests run on the survey.

Collecting results typically takes more time and depends on the communication channels available to distribute the survey. Response rates can vary from 1 to 20 percent on a survey sent to existing customers, depending on the level of customer engagement, so large numbers of target customers and a good deal of time may be required to collect data.

Analyzing the data can take 4-8 hours, depending on the length of the survey, the number of respondents, and the quality of responses. As answers are free text, a large amount of reading, transcribing, and synthesizing may be required.

How to

Preparation:

  1. Write screening questions.
    • These are typically closed-ended questions that help identify if the respondent is in the desired target segment (e.g., “How old are you?”).
    • A few leading questions can be placed in a survey to identify “professional survey respondents” who will lie to be included in a survey or have a chance to participate in a follow-up research project for cash.
  2. Write questions.
    • Questions should be non-leading and non-hypothetical.
    • Asking for anecdotes or historical information can generate more concrete insights.
    • Conduct comprehension tests on survey questions.
    • This ensures that the questions are being correctly interpreted, and can often reveal leading questions.

Distribution:

  1. Surveys can be sent out via any method to the target audience.
  2. Typical distribution methods include:
    • Social media
    • Email
    • Website pop-ups
    • Regular mail
    • Telephone
    • SMS

Interpreting Results

An open-ended survey is a generative research technique, and as such, be careful to interpret any input as simply ideas, not as a vote from the customer. The data is qualitative in nature.

Because surveys are flexible, easy to write, and easy to deploy, they are more likely to be misunderstood and misused. Surveys are often a default research method when researchers do not feel they have the time to conduct ethnography or customer interviews. They are often highly favored in a corporate setting because a large number of respondents may be considered statistically significant, even if the survey responses are qualitative.

Open-ended surveys are sometimes combined with closed-ended surveys, making it tempting to spend extended periods of time analyzing the data and looking for correlation that will draw a definitive conclusion to act upon. This tendency to use the data to drive a firm conclusion even when the data is generative in nature is the biggest argument for avoiding surveys at all costs.

A typical debrief method to analyze the generative data is to read each answer and transcribe salient points on post-it notes for a sorting exercise. Patterns can then be more easily identified.

For surveys that specifically solicit suggestions from users, the entire list of suggestions may be added to a repository for later analysis.

In the case of very large data sets, algorithmic tools such as sentiment analysis or word clouds can give additional quantitative insight, but should be used to supplement the qualitative insights, not replace them.

Potential Biases

  • Selection bias: Researchers will often fixate on qualitative comments they agree with and ignore other comments.
  • Sampling bias: Although the sample may not match the general population to be surveyed, if the data is taken as generative and not evaluative, this bias is less relevant. Any ideas generated still must be validated by an evaluative method.

Field Tips

  • You can run an open-ended survey once you know the best questions to ask. Talk to your customers to figure out the right questions. - @TriKro
  • Got a tip? Add a tweetable quote by emailing us: [email protected]

Case Studies

Tools

References

Relevant resources

Workshop

Effective Survey Design

Surveys are one of the best tools for an innovator -- and one of the worst.