Introduction
During the survey design process, you define “Cutoff rules” to direct Ask when to stop inviting people to participate in the survey (see Information: Cutoff Rules - link). You can set different types of cutoff rules, such as continuing the survey until there are 5,000 successful interviews, or until there are 400 interviews from various sub-groups of the population.
Defining the cutoff rules is one of the most important decisions you will make as a survey designer. If you set the wrong cutoff rules, you may have too much or too little data collected. Incorrect cutoff rules can also end up costing unnecessary time and money. This document provides some best practices in setting cutoff rules.
Guidelines for Setting Cutoff Rules
Do set cutoff rules for most surveys.
In Ask, cutoff rules are optional. If you do not set a cutoff rule, then the survey will run indefinitely - i.e., it will continue collecting data and will not stop. This may be appropriate in some surveys. However, most surveys will use a cutoff rule. Generally, using cutoff rules is recommended unless there is a valid reason not to do so.
Do work with a statistician to calculate the appropriate cutoff rules.
Cutoff rules affect how the number of successful interviews will be available when someone analyzes the data. A statistician can help determine how many completed interviews are needed. The larger the number, the more precision, but the more money and time are needed to collect the data. For example, in the NCD Mobile Phone Survey, a standardized sample design suggests the number of completed interviews for different quota groups.
Do be mindful of how cutoff rules affect cost.
The Ask tool itself is free. However, it costs money to send Short Message Service or Interactive Voice Response calls to respondents via mobile network operators. Best practice is to first generate the cutoff rules from a statistical perspective, then modify the cutoff rules in light of available budget.
Do remember that cutoff rules can operate on both completed and partial interviews.
By default, the Ask tool sets cutoff rules based on completed and partial interviews. You can also check an option to allow partial interviews to count toward cutoff rules.
Don't try to define quota groups until the questionnaire has been designed.
Remember, you can define quota groups only based on questions already in the questionnaire.
Don't include too many quota groups.
Quota cutoff rules are great for ensuring that the survey reaches different groups. However, quotas can be problematic if you have too many of them. For example, if the age quotas are too narrow (e.g., 45-49 years, 50-54 years, 55-59 years), filling up some of the quotas may be difficult. Ask will continue to run until all the cutoff rules are met, which may cost unnecessary time and money. In general, we recommend a maximum of 6 quota groups. Each quota group can be a fairly wide demographic group (e.g., women 18-35 years; men 18-35 years; women 34-49 years; men 34-49 years).