3.8 Open-Ended Survey

In Brief

Description

Open-ended surveys let respondents answer in their own words rather than picking from a list. The output is free text — sentences, paragraphs, or fragments — collected at scale across a target audience, then read for themes rather than tallied. They are the right choice when you don’t yet know what the answer categories should be: a closed-ended list can only return the options you already thought of, so it cannot surface a problem or phrasing you didn’t anticipate.

Open-ended and closed-ended surveys are not interchangeable — they answer different questions. Use open-ended when you’re still discovering: when the categories don’t exist yet, when you need respondent language verbatim, or when an unexpected theme would change your direction. Use closed-ended when you already know the categories and need to measure how big each one is. Running them in the wrong order — closed-ended before open-ended — locks you into your existing assumptions and hides the surprises you ran the survey to find.

A 200-respondent survey produces 200 or more paragraphs of free text to read, which is where teams stall. AI handles the volume — a first-pass thematic analysis takes minutes — but a human still has to interpret the themes and decide what they mean for the business.

How to

Prep AI Prompt

1. Write screening questions. Filter for the people you actually want to hear from.

   • These are typically closed-ended questions that help identify if the respondent is in the desired target segment (e.g., “How old are you?”).
   • Add a few trap questions to catch “professional survey respondents” who lie to qualify for a survey or a paid follow-up study.

2. Write the questions. Keep them non-leading and grounded in real behavior.

   • Questions should be non-leading and non-hypothetical.
   • Asking for anecdotes or historical information generates more concrete insights.
   • Conduct comprehension tests on survey questions to ensure they’re correctly interpreted.
   • LLMs can help draft question candidates and generate comprehension test variants to check for bias before deployment — but always review AI-generated questions yourself, as LLMs tend toward leading or hypothetical phrasing that sounds neutral but subtly biases responses.

   Examples of good open-ended questions:

   • “Describe the last time you tried to [task]. What happened?” (surfaces real behavior, not hypotheticals)
   • “What’s the most frustrating part of [process] for you?” (identifies pain points in their words)
   • “If you could change one thing about how you [activity], what would it be?” (reveals priorities)
   • “What have you tried so far to solve this problem?” (maps existing alternatives)

   Avoid: “Would you use a product that does X?” (hypothetical), “Don’t you think X is a problem?” (leading), “How satisfied are you?” (closed-ended — save for a Closed-Ended Survey).

3. Pilot with 5–10 people from your target segment. Catch confusing questions before you spend your list on them.

   • Run the draft survey on a small group before launch. You’re checking for questions that confuse people, missing context, and how long the survey actually takes (aim for under 5 minutes).
   • If a respondent asks what a question means, that question needs rewriting — don’t explain it for them.
   • If most pilot answers come back one-word or “N/A,” the question is failing to get a reaction. Rewrite or cut.

4. Pick a distribution channel. Reach your audience where response quality holds up.

   • Match the channel to your audience. Common options:
     • Social media
     • Email (existing list)
     • Website pop-ups or in-product surveys
     • Regular mail
     • Telephone
     • SMS
   • Email to an existing list typically produces the highest response rates for open-ended formats; cold or paid panels skew toward shorter, lower-effort answers because respondents have less context for the topic.

Analysis AI Prompt

Treat each response as an idea to explore, not as a vote. The data is qualitative: themes point you toward what to investigate next, they do not measure how many people hold a view.

Because surveys are easy to write and deploy, they are easy to misuse. Teams reach for them as a default when they feel they lack time for interviews or observation, and a large respondent count tempts a corporate audience to treat qualitative answers as statistically significant. The pull is strongest when an open-ended survey runs alongside a closed-ended one: the numbers invite you to hunt for correlations and declare a definitive conclusion the free text cannot support. That overinterpretation is the most common reason teams are warned off surveys.

To theme the responses, read each answer and pull the salient points into groups so patterns surface. AI does the first pass: paste the responses in and ask for the top five themes with a representative quote each, or use a dedicated analysis tool to code responses into themes at volume, score sentiment, and surface unexpected patterns. Let AI generate the initial coding pass, then review, split, merge, and validate the categories against what you already know about your audience.

Expected response rates and item-nonresponse. For email-distributed surveys to your own list, expect 5–20% response rates. With fewer than 30 responses, treat themes as hypotheses to explore in interviews, not conclusions to act on. Open-ended questions also produce higher item-nonresponse than closed-ended ones: skip rates commonly average around 18% (versus 1–2% for closed-ended), and can run anywhere from a few percent to over half on a single question. If a single question’s skip rate is well above the average for your survey, the question itself is the problem — too long, too hard, or unclear.

When a survey specifically asks for suggestions, keep the full list of suggestions in a repository for later analysis.

For very large data sets, algorithmic tools such as sentiment analysis or word clouds can add quantitative insight, but use them to supplement the qualitative findings, not replace them.