Summary
Audience: Non-technical staff assessing readiness to run surveys at moderate scale.
When you have a Voice channel set up with an MNO or aggregator and you want to assess readiness for a Pre-test or Scaled survey, run the following tests.
These are not the types of tests that validate content accuracy or understandability of the questionnaire, or validates quality of translations, which are commonly called "cognitive tests".
Test 1: Basic Channel Validation
This test validates the expected minimum functionality of the channel. The outputs will change depending on each channel's status, so check with technical support to see if your expectations match the setup.
Preparation
You must have:
- A simple questionnaire with some high-quality introductory audio you can listen for at least 30 seconds, and a yes/no question,
- Knowledge about which MNO your channel is connected to (e.g. is it Airtel? MTN?), denominated "MNO X" in the next steps.
- Knowledge if you are to expect calls from a shortcode, and which number that is (usually a 4- to 6- digit number such as "7272"),
- A CSV file with At least 2 phones for each MNO, and optionally a landline,
- Some volunteers to help pick up all the phones in time,
- Understanding of the USSD codes to check your phone credit, for each MNO. See: How To: Check Credit on Your Phone.
Test Steps
- Check credit on all respondent phones as applicable,
- Create a survey with a simple questionnaire with an multiple-choice option question,
- Select Voice Mode, X Channel,
- Use respondent CSV file as described above,
- Verify Schedule is OK and that it includes your current time and it is in the right time zone,
- Instruct your volunteers to press all "1" or "2" in the yes/no question.
- Run Survey (You may need some volunteer help picking up all phones in time),
- Stop survey when done,
- Inspect results in Surveda and interview listeners for voice quality,
- Check credit on all respondent phones as applicable.
- Optional: Call back the incoming number from respondent phones.
You can run all the tests in one run or multiple runs, depending on the time available, volunteers available, etc.:
Must-Have Results
These are the basic minimum results you should expect if the channel is working:
- Calls are received by all respondents,
- More than one call is received simultaneously, at the same time,
- Audio sounds OK - no garbling or strange delays,
- DTMF answers are processed ok (pressing "2" results in appropriate answer in Surveda),
- Incoming calls do not deplete credit from respondents,
- Numbers for completed surveys appear in the incentives file.
Nice to Have Results:
The following results are not required for the channel to be used for surveys, and may or may not be expected depending on contract or technical constraints. You should not expect these to work and will not be be simultaneously feasible if only some MNOs have the channels configured:
- Call comes from your assigned shortcode as (e.g. caller ID shows "7272") Note: This is not possible in all countries,
- For respondents using cell phones with SIM cards from MNO X: Call comes from shortcode. Note: This is not possible in all countries and requires the MNO to configure caller IDs adequately,
- For respondents using cell phones with SIM cards NOT from MNO X: Call comes from shortcode. Note: This is not possible in all countries and requires the MNO to configure caller IDs adequately, and sometimes shortcode to registered in all MNOs,
- Calling back to the caller ID number will play an MOH explanatory audio when the respondent belongs to MNO X. Note: this is not possible in all countries,
- Calling back to the caller ID number will play an MOH explanatory audio when the respondent belongs to MNO X,
- If call-backs to the number have worked, respondent phone credits shouldn't have been used.
Test 2: Optional Channel Information Gathering
This test helps set expectations about the status codes obtained from Surveda when dialing numbers that are non-responsive, erroneous, fax machines, invalid, etc. It is not a validation test, rather it helps gather information that could be useful in statistical analysis of the results of a larger survey.
Preparation
Follow the same steps as the test above, but instead of incorporating valid numbers, include one valid number and one or two of each of the following (test with what you have and know):
Data Management and Analysis Manual Ch 2: Data Management and Quality Assurance Processes.
Note: You may need to perform multiple test runs and collect the information somewhere shared (consider submitting it to this portal)
- A landline that won't pick up,
- An inactive landline (a number that looks ok but is not assigned to anything -try by calling),
- An inactive mobile phone of each MNO,
- An invalid mobile phone of each MNO (eg. 096-01000000),
- A fax line,
- A pager,
- A mobile phone that will ring but won't pick up (one from each MNO),
- A mobile phone that is off (one from each MNO),
- A designated "busy" phone that is will be on another call while the survey runs (one for each MNO and a landline),
- A mobile phone that will ring and pick up but the respondent will hang up in the 10th second (one from each MNO),
- A mobile phone that will remove the battery in the middle of the survey,
- etc.
Results
The purpose of this optional test is to gather information about how the behavior of phone lines in different situations affects status codes obtained via Surveda. Results will vary country by country, and may vary MNO by MNO, so if it is relevant to statistically compare non-responding or cutoff codes for surveys performed through channels of different MNOs or aggregators, it may be good to gather this information for each channel.
More Information
How To: Configure, Run, and Monitor a Survey
Data Management and Analysis Manual Ch 2: Data Management and Quality Assurance Processes
How To: Check Credit on Your Phone