Final Disposition Codes

A.2    Margin of Error for Key Survey Estimates

Background

Measures of statistical precision of estimates are incorporated here to assist with analysis.  For example, a sample mean will be distinctive for various samples of a similar size taken from the same population, resulting in mean estimates that are all slightly different than the true population parameter. Sampling error is a major behind the distinction between an estimate and the true population parameter. The sampling methodology recommended as a part of the NCD Mobile Phone Survey allow analysts to measure of the precision of estimates. To characterize these measures, the symbol P signifies the parameter being evaluated (e.g., the prevalence rate of persons who always or often eat processed foods that are high in salt). represents the estimate of the population parameter.

An estimate of statistical precision is the variability of the estimate, composed as V(P ̂). The variability of estimates is a marker of how much sample estimates would differ among all the conceivable samples drawn from the same sample design. The standard error of the estimate is .  The relative standard error of the estimate is:

.

measures accuracy with respect to effect size.  As a result, it is standardized (a.k.a., without unit) which makes it useful for comparing indicators.

The measure of precision countries should report in the NCD Mobile Phone Survey is the MOE, characterized as , where Z is a measure of the level of certainty for the measure and is the standard error of . is interpreted as:

We are 95% confident that the reported value () is within the amount MOE() of .

NCD Mobile Phone Survey analysts are urged to report the value of MOE() for all key estimates.

Data Sources

The final weighted data file used for analysis should be used for these calculations.

Computational Software

Estimates of population parameters and variability must account for sample design. Estimates must be weighted, and incorporate the use of stratification, naturally occurring clustering, without-replacement sampling, and sample weights. Failure to do so contributes to biased estimates as well as inappropriate interval estimates and tests of significance. Therefore, NCD Mobile Phone Survey country analysts are strongly urged to use analysis software that allows one to fully account for the sample design used to produce the survey data.  It implies the use of software that incorporates a widely accepted approach to variance estimation.

Two statistical organizations globally have developed computer software to analyze data from complex samples like the NCD Mobile Phone Survey. These software programs will not only produce survey estimates (i.e., ), but they can also produce estimates of precision (i.e., usually either V() or SE()) that appropriately account for key design features in the NCD Mobile Phone Survey, namely naturally occurring clustering, the use of stratification, and varying selection probabilities (i.e., sample weights).  Refer to Section 3.2 in the Data Management and Analysis Plan Manual for a description of these statistical software packages.

The NCD Mobile Phone Survey analysts provide technical assistance for use of the following statistical software packages: Stata and R.

More Information

For additional details on the statistical definitions provided earlier, refer to the Encyclopedia of Survey Research Methods by Paul Lavrakas. The website sponsored by the Survey Research Methods Section of the American Statistical Association also has additional software information (available from URL: www.hcp.med.harvard.edu/statistics/survey-soft).

Computation

Output from the software listed earlier will report a value of , as well as its estimated variance, denoted by V(), or its standard error, written as SE(). From these reported values, one can compute the estimate MOE for , as

.

Interpretation

The value of MOE() reported for is interpreted as follows:

We are 95% confident that the estimated value () is within the amount MOE() of .

A.3    Estimates of Sampling Errors

The respondents in the NCD Mobile Phone Survey in [country] make up just a single sample of all the conceivable samples that could have been chosen from the same population, utilizing the sample sampling design. Sampling errors are a measure of the precision between every single conceivable sample. Despite the fact that the degree of precision is not known precisely, it can be evaluated from the survey data.

The following sampling error measures are presented for each of the selected indicators:

• Value (R). Weighted prevalence estimate of the indicator.
• Standard error (SE). Sampling errors are measured by the standard errors for a particular estimate or indicator. Standard error of an estimate is the square root of the variance of that estimate.
• Sample size (n). Total number of observations used to calculate the prevalence estimate (R).
• Design effect (Deff). The design effect is the ratio of the actual variance of an indicator, under the sampling method used in the survey, to the variance calculated under the assumption of simple random sampling. A Deff value of 1.0 indicates that the sample design is as efficient as a simple random sample, while a Deff value above 1.0 indicates the increase in the standard error due to the use of a more complex sample design. In general, for a well-designed survey, Deff usually ranges from 1 to 3. It is common, however, for Deff to be much larger, up to 7 or 8.
• Relative standard error (RSE). Also known as coefficient of variation (CV), this is the ratio of the standard error to the value of the indicator.
• MOE. Margin of error is calculated as the product of the desired confidence measure and the standard error of the estimate. The level of confidence is usually based on a value (Z) of the standard normal distribution. For example, for a 95% level of confidence, use Z = 1.96.
• Confidence limits (R±1.96SE). Calculated to show the interval within which the true value for the population can be reasonably assumed to fall. For any given statistic calculated from the survey, the value of that statistic will fall within a range of plus or minus two times the standard error of the statistic in 95% of all possible samples of identical size and design.

Calculation of Standard Error

The NCD Mobile Phone Survey [year] sample is the result of a two-phase stratified design, so it is necessary to use complex formulae for estimating standard errors. For the calculation of standard errors from NCD Mobile Phone Survey [country] data, [statistical software version] was used. The Successive Difference Replication (SDR) and model-assisted variance estimation methods should be used for survey estimates. Analysts can use the output from the appropriate statistical software package to obtain the standard errors.  Currently, there are only two software packages that support a two-phase stratified design.  Stata employs the SDR method with the sdr option in the svy module.  R employs the model-assisted method with the twophase function in the survey package, as well as within the functionality of the osDesign package.

The results are presented in this appendix for the country as a whole and for sex. For each variable or indicator, the type of statistic (mean, proportion, or rate) and the base population are given in Table A.4. In addition to the standard error (SE) described earlier, Table A.5 includes the value of the estimate (R), the sample size, the design effect (Deff), the relative standard error (RSE, MOE, and the 95% confidence limits (R±1.96SE) for each variable or indicator.

A.4    List of Indicators for Sampling Errors, NCD Mobile Phone Survey [country] [year]

A.5    Example Table for Reporting Sampling Errors