Lecture 2 -- Chapter 2 PDF

**Lecture 2 -- Chapter 2** **2.4 Sources of Errors in Surveys** Survey errors can be divided into two major groups: 1. ***Errors of nonobservation***, 2. ***Errors of Observation**.* Errors of nonobservation can be classified as: 1. Sampling error 2. Coverage error 3. Nonresponse error...

**Lecture 2 -- Chapter 2** **2.4 Sources of Errors in Surveys** Survey errors can be divided into two major groups: 1. ***Errors of nonobservation***, 2. ***Errors of Observation**.* Errors of nonobservation can be classified as: 1. Sampling error 2. Coverage error 3. Nonresponse error Errors of observation can be classified as: 1. Interviewer error 2. Respondent error 3. Instrument, or method of data collection error. 1. **Sampling error:** Generally, the data observed in a sample do not exactly reflect the data in the population from which that sample was selected, even if the sampling and measuring are done with extreme care and accuracy. This deviation between an estimate from a sample and the true population value is the *sampling error* that is produced simply because this is a sample and not a census. Sampling error can be measured theoretically and estimated from the sample data for probability samples. It is important to note that sampling error can be reduced by good survey designs and appropriate choice of sample size. Thus, the investigator has some control over this component of error. 2. **Coverage error:** Coverage error occurs when the target population does not coincide with the population actually sampled. The source of the coverage error may be an inadequate sampling frame or flaws in the implementation of the data collection. Coverage error results because of undercoverage and overcoverage. Undercoverage occurs when members of the target population are excluded. Overcoverage occurs when units are included erroneously.  Example of common sources of coverage error: - Telephone surveys exclude people without telephones - Most telephone surveys exclude cell phone users - Internet surveys exclude non-Internet users **Nonresponse error:** Nonresponse Error occurs when some of the respondents you select in your sample don't respond. Nonresponse arises in one of three ways: - the inability to contact the sampled element (person or household, for example), - the inability of the person responding to come up with the answer to the question of interest, or - refusal to answer. Data must be collected from precisely those elements that were selected by the randomization scheme used in the design of the survey. An interviewer must not substitute a next door neighbor who just happens to be home for the person actually selected for the sample. This type of substitution might lead to a survey that is biased because too many families with children or too many retired people or too many people who work at night are being interviewed. In addition to these obvious biases, haphazard substitutions alter the probabilistic structure of the design and may make it impossible to estimate the sampling error. The inability of the interviewed person to answer the question of interest is a serious problem, particularly questions that deal with facts. A question on opinion can have a "don't know" option, and the survey design can account for a certain percentage being in this category. The most serious aspect of the nonresponse problem today is refusal to answer. Perhaps because of the proliferation of surveys, because of fear related to increases in crime, and, no doubt, because of a variety of other reasons, people are refusing to answer survey questions in ever-increasing numbers. **Errors of observation** **Interviewer error:** *Interviewers* have a direct and dramatic effect on the way a person responds to a question, as previously mentioned. Reading a question with inappropriate emphasis or intonation can force a response in one direction or another. Most people who agree to an interview do not want to appear disagreeable and will tend to side with the view apparently favored by the interviewer, especially on questions for which the respondent does not have a strong opinion. Friendly interviewers have more success, of course, than the overtly forceful ones. How gender issues affect interviews is not clear, but male interviewers get a higher rate of cooperation from male respondents than do female interviewers. In general, interviewers of the same gender, racial, and ethnic groups as those being interviewed are slightly more successful. **Respondent error:** *Respondents* differ greatly in their motivation to answer correctly and in their ability to do so. Each respondent must understand the entire question and be clear about the options for the answer. This means that questions must be clearly phrased and the questionnaire should not be too long because people will quickly tire of the interview. Obtaining an honest response to sensitive questions is particularly difficult and may require special techniques. An attempt to place response errors into categories suggests that most are due to either - recall bias (the respondent simply does not remember correctly), - prestige bias (the respondent exaggerates a little on income or hunting success), - intentional deception (the respondent will not admit breaking a law or has a particular gripe against an agency), - incorrect measurement (the respondent did not understand the units and reported feet instead of inches or did not understand the definition of *children* and reported grandchildren as well). **Instrument, or method of data collection error:** The incorrect measurement issue is related to the *measurement instrument* as a source of error. In any measurement question, the unit of measurement must be clearly defined. Inaccurate responses are often caused by errors of definition in survey questions. Some examples are the following: \(1) The word *children* must be clearly defined. \(2) What does the term *unemployed* mean? Should the unemployed include those who have given up looking for work, teenagers who cannot find summer jobs, and those who have lost part-time jobs? \(3) Does *education* include only formal schooling or technical training, on-the-job classes, and summer institutes as well? Items to be measured must be precisely defined and be unambiguously measurable. **Reducing the Errors in the Survey:** Both errors of nonobservation and errors of observation can seriously affect the accuracy of a survey. Errors cannot be eliminated from a survey, but their effects can be reduced by careful adherence to a good sampling plan. Some major points in reducing survey errors are presented next. **Callbacks** **Rewards and incentives** **Trained interviewers** **Data Checks** **Questionnaire construction** **2.6 Planning a Survey** Generally, the following steps are followed in planning a survey: **1. *Statement of objectives.*** State the objectives of the survey clearly and concisely and refer to these objectives regularly as the design and the implementation of the survey progress. Keep the objectives simple enough to be understood by those working on the survey and to be met successfully when the survey is completed. **2. *Target population****.* Carefully define the population to be sampled. If adults are to be sampled, then define what is meant by *adult* (all those over the age of 18, for example) and state which group of adults are included (all permanent residents of a city, for example). Keep in mind that a sample must be selected from this population and define the population so that sample selection is possible. **3. *The frame.*** Select the frame (or frames) so that the list of sampling units and the target population show close agreement. Keep in mind that multiple frames may make the sampling more efficient. For example, residents of a city can be sampled from a list of city blocks coupled with a list of residents within blocks. **4. *Sample design.*** Choose the design of the sample, including the number of sample elements, so that the sample provides sufficient information for the objectives of the survey. Many surveys have produced little or no useful information because they were not properly designed. **5. *Method of measurement****.* Decide on the method of measurement, usually one or more of the following methods: personal interviews, telephone interviews, mailed questionnaires, or direct observations. **6. *Measurement instrument.*** In conjunction with step 5, carefully specify how and what measurements are to be obtained. If a questionnaire is to be used, plan the questions so that they minimize nonresponse and incorrect response bias. **7. *Selection and training of fieldworkers.*** Carefully select and train fieldworkers. After the sampling plan has been clearly and completely set up, someone must collect the data. Those collecting data, the fieldworkers, must be carefully taught what measurements to make and how to make them. Training is especially important if interviews, either personal or telephone, are used because the rate of response and the accuracy of responses are affected by the interviewer's personal style and tone of voice. **8. *The pretest.*** Select a small sample for a pretest. The pretest is crucial because it allows you to field-test the questionnaire or other measurement device, to screen interviewers, and to check on the management of field operations. The results of the pretest usually suggest that some modifications must be made before a full scale sampling is undertaken. **9. *Organization* of *fieldwork*.** Plan the fieldwork in detail. Any large-scale survey involves numerous people working as interviewers, coordinators, or data managers. The various jobs should be carefully organized and lines of authority clearly established before the survey is begun. **10. *Organization of data management****.* Outline how each datum is to be handled for all stages of the survey. Large surveys generate huge amounts of data. Hence, a well-prepared data management plan is of utmost importance. This plan should include the steps for processing data from the time a measurement is taken in the field until the final analysis is completed. Aquality control scheme should also be included in the plan in order to check for agreement between processed data and data gathered in the field. **11. *Data analysis and report writing.*** Outline the analyses that are to be completed. Closely related to step 10, this step involves the detailed specification of which analyses are to be performed. It may also list the topics to be included in the final report. If you think about the final report before a survey is run, you may be more careful in selecting items to be measured in the survey. If these steps are followed diligently, the survey will be off to a good start and should provide useful information for the investigator.