Closed-Ended Survey

In Brief

Closed-ended surveys are structured questionnaires that ask participants to choose between fixed, multi-choice answers such as yes/no or A/B/C/D. They are often used to rank the most important issues for participants or segment the population. Surveys can be repeated over time to identify trends. The output is structured quantitative data which can be analyzed with statistical techniques.

Description

Closed-ended surveys force participants to pick from a fixed set of answers. Every respondent sees the same options, so the output is structured data you can count and compare.

Use them when you have “known unknowns” — you’ve done generative research, you have several hypotheses, and now you need numbers to decide between them. Don’t use them to discover problems you haven’t heard of yet. Open-ended methods are better for that.

Surveys can be run once to make a decision or repeated over time to track a trend. They scale cheaply: the same instrument works for 50 respondents or 5,000.

How to

Prep AI Prompt

1. Write your goal in one sentence.

Keep it focused and don’t try to learn everything at once. A survey built to segment a market looks nothing like one built to measure satisfaction. If you can’t state the goal in one sentence, you’re not ready to write questions.

Examples:

• “Determine which of four pain points is most common among mid-market CFOs.”
• “Measure whether satisfaction improved after the Q3 onboarding redesign.”

2. Pick a screener question.

The first question should filter out people who aren’t in your target population. Put it at the top and end the survey for anyone who doesn’t qualify. Try to avoid multiply screeners.

Are you currently responsible for purchasing software for your team? Yes / No

(If No → end survey)

3. Choose your question types.

Each type produces a different kind of data. Use the one that matches what you need to learn.

Dichotomous (yes/no) — Splits the population into two groups. Fast to answer. Use when you need a clean binary.

Do you currently use a project management tool? Yes / No

Multiple choice — Identifies which category a respondent falls into. Use for segmentation or feature preference.

What is your primary role?

• Founder/CEO
• Product Manager
• Engineer
• Designer
• Other

Likert scale — Measures degree of agreement or satisfaction on a fixed scale. Use to gauge intensity, not just direction.

“I find it easy to prioritize my daily tasks.” Strongly disagree / Disagree / Neutral / Agree / Strongly agree

Forced ranking — Makes respondents prioritize. Unlike multiple choice, it forces trade-offs. Use when you need relative importance, not just preferences.

Rank from most to least important when choosing a SaaS tool:

1. Price
2. Ease of use
3. Customer support
4. Integrations

Matrix — Applies the same scale across several items. Efficient but fatiguing. Keep it to five rows or fewer.

Rate each department’s data literacy (1 = Low, 5 = High):

• Marketing
• Product
• Sales
• Engineering

Hybrid (closed + “Other”) — Add a free-text “Other” option when you aren’t sure your answer list is complete. If more than 10% of respondents choose “Other,” your options are wrong. Go back and fix them.

4. Write the questions.

Rules:

• One concept per question. Don’t ask “Is the product fast and reliable?” — that’s two questions.
• No hypotheticals. “Would you pay for X?” is worthless. People don’t know.
• No leading language. “Don’t you agree that…” is not a question.
• Include “N/A” or “Don’t know” where appropriate. Forcing an answer where none exists creates noise.
• Keep the total survey under 15 questions. Completion rates drop sharply after that.

5. Order the questions deliberately.

• Start with the screener.
• Put easy, non-threatening questions next to build momentum.
• Place sensitive or complex questions in the middle, after trust is established.
• End with demographic questions (role, company size, etc.) — these feel impersonal and signal the survey is nearly done.
• If order effects matter, randomize question sequence across respondents. Most survey tools support this.

6. Pilot with 5–10 people.

Run the survey on a small group before you launch. You’re checking for:

• Questions that confuse people
• Missing answer options
• How long it actually takes (aim for under 5 minutes)
• Whether the data you get back answers your stated goal

7. Choose a distribution channel.

Match the channel to your audience:

• In-app or website exit survey — Best response rates. Use for existing users. Tools like Sprig or Hotjar embed directly.
• Email — Good for existing customers or mailing lists. Expect 10–30% response rates. Keep the subject line short and specific.
• Social media or community posts — Cheap reach but self-selecting. Useful for early-stage validation when you don’t have a user base yet.
• Paid panels — Services like Pollfish or Prolific recruit respondents matching your criteria. Costs $1–5 per response. Use when you need a specific demographic you can’t reach organically.
• In-person — Pen and paper at events, trade shows, or retail locations. High completion rates but small sample sizes.

8. Set a target sample size.

You need enough responses to trust the results. Rules of thumb:

• For simple yes/no splits: 100 responses minimum.
• For comparing subgroups (e.g., role A vs. role B): 30+ per subgroup.
• For detecting small differences between options: 300+.
• Use a sample size calculator to get a precise number. Plug in your expected effect size, confidence level (usually 95%), and population size.
• If your sample will be small (under 50): You can still run the survey. You won’t get statistical significance, but you will get directional signals — which options cluster together, which segments lean one way. Present findings as “early indications,” not conclusions. Small-sample surveys are most useful when paired with follow-up interviews to pressure-test the patterns you see.

Analysis AI Prompt

1. Clean the data first.

• Remove incomplete responses (respondents who dropped out partway through).
• Remove speeders — anyone who finished in less than one-third of the median completion time was clicking randomly.
• Check for straight-liners: respondents who selected the same answer for every question. Remove them.

2. Start with frequencies.

For each question, count how many respondents chose each option. Calculate percentages. This is your baseline. Look at it before doing anything fancy.

3. Cross-tabulate.

Cross-tabulation means breaking responses down by subgroup (e.g., role, company size, or answers to other questions). A preference that looks evenly split overall might be 80/20 within a specific segment. That’s where the insight is. You can do this in a spreadsheet with pivot tables, or ask an AI to do it for you.

4. Check distributions.

Don’t just look at averages. Look at the shape of how answers are spread out:

• Bimodal (two peaks) — You may have two distinct segments responding differently. Investigate.
• Heavily skewed (most answers bunched to one side) — The average is misleading. Report the median instead.
• Uniform (answers spread evenly) — Respondents are guessing or the question doesn’t discriminate. The question may be poorly worded.

5. Test for significance.

You don’t need a statistics background to do this. Export your data as a CSV, paste it into an AI tool, and ask it to run the appropriate test. The key tests and when to ask for them:

• “Run a chi-square test” — When you’re comparing categories across groups (e.g., “Did segment A prefer option 1 more than segment B?”). This tells you whether the difference is real or just random variation.
• “Run a t-test” (two groups) or “Run an ANOVA” (three or more groups) — When you’re comparing average scores across groups on Likert-scale or rating data.
• “Check for correlations” — When you want to know whether answers to two questions move together (e.g., do people who rate ease-of-use highly also rate satisfaction highly?).

The AI will return a p-value. If it’s below 0.05, the pattern is statistically significant — meaning it’s unlikely to be noise. If it’s above 0.05, you can’t trust the pattern yet. You may need more responses.

When an AI returns an analysis, open a new session and ask it to doublecheck and critique the analysis to avoid hallucinations.

6. Flag problems in the instrument.

• Questions where 80%+ chose the same answer told you nothing. The question was too obvious or too leading.
• Questions with high “N/A” or “Don’t know” rates were asked of the wrong audience.
• If “Other” exceeded 10% on a multiple-choice question, your options were incomplete.