RMP470S lecture 5 - Survey research methods.pdf

Advanced Diploma Research Methodology and Project (RMP470S) Survey research methods  Dr N. Dlangamandla  Email: [email protected]  Why undertake a survey? Engineering research only valuable when it directly or indirectly...

Advanced Diploma Research Methodology and Project (RMP470S) Survey research methods  Dr N. Dlangamandla  Email: [email protected]  Why undertake a survey? Engineering research only valuable when it directly or indirectly contributes to the improvement of the human conditions. One form of research is to seek information about and/or feedback from people about the outcomes or proposed outcomes of the research This could be a product or service Assessment of the usability of a product or service can be the subject of research. Failure to consult users of developing technology may restrict it's use in society  Why undertake a survey? Example: Developer of new technology by addressing the following questions: Is the technology limited to a particular age group, ethnic group, or group with disabilities? Will technology cause unintentional harm to users and the environment? Is it possible that the technology can cause injury, disability and even death in the very worst cases?  Why undertake a survey? One method to review these questions is to seek feedback from the potential users and the public at large. There are also design challenges for devices that can cause potential harm from intended or unintended users. The only method for obtaining some of this information in a rational, controlled manner from a representative sample population is through the use of a well- constructed, scientifically based survey  Why undertake a survey? Two types of engineering investigations that require feedback from user groups or the general population: Defining characteristics of humans –physical, mechanical, chemical, genetic and physiological parameters including impairment Commonly described in terms of anthropometric, biomechanical and biochemical measurements. Defining the opinions of humans –what they think about objects, engineering plans and outcomes and the usability of objects and systems. Commonly describes in terms of social, cultural, religious and political aspects of human life. This can be described as a usability survey.  Why undertake a survey? Product development is best undertaken using concurrent engineering principles Researchers must be aware of the human limitations in using and operating the technology and the basic requirements underlying the survey research methods. This includes reporting methods and the educational level, history and cultural aspects of the community.  Survey requirements Population surveyed must be sufficiently large so that appropriate statistical analyses can be undertaken. Time allocated to complete the survey must be relatively short to ensure recent events do not impact on the survey outcomes. Population must be sufficiently large to ensure that individual identities cannot be inferred from the statistical data presented. Population must be sufficiently diverse to ensure that population groupings are not subjected to adverse stereotyping. Participation is voluntary and there is no mandatory requirements that all questions/sections of the survey are completed The survey results are presented to the population surveyed in an understandable form.  Ethics approval If a research team administers a survey, then there is a possibility of bias and confidentiality can be compromised. Many surveys are commissioned to be carried out by independent survey organisations. This is particularly true for companies seeking to understand the thoughts of employees.  Ethics approval Example: It will be difficult for workers to give honest answers when asked to comment on working conditions or their salary when directly confronted by the boss. It is difficult for students in a class to comment on the classroom teacher if the teacher is doing the actual survey. The results will be biased as the person asking the question has power and influence over the individual.  Ethics approval As a survey involves seeking the opinions or personal details of others, ethics approval to conduct the research is mandatory. Most journals will not allow publication without proof of The requirement of ethics approval is to ensure that the cohort of people whose opinions are sought, are not inadvertently damages (physically, mentally or reputationally)  Ethics approval Most organisations have a research ethics committee with members who are trained in medicine, law, psychology and research practice. Some are registered by a government authority. An audit may be conducted during or after the research to ensure that the research was conducted responsibly: That all participants signed a consent form That the research outcomes have been made available to the participants  Ethics application The application for ethics approval requires a justification for the research (what is the aim and why it is important) and the submission of 3 documents: Information sheet This describes the project aims, methods, requirements of the participants, personal details information and includes the names and contact details of the researchers. This sheet is passed to the participants before the measurements begin. Participants are encouraged to keep the sheet. Consent form The participants sign and date the sheet. The sheet is retained by the researcher. Each participant is assigned a number. Survey sheet This is the sheet which is completed by the research team and/or each participant. Only the participant number is written on this sheet. The name of the participant is not included. This allows the information to be stored separately form the consent forms.  General survey guidelines There are many textbooks which describe strategic approaches to survey design. There are semi-automated tools available on the web Google survey tool SurveyMonkey Rather than asking questions, surveys should seek the reaction of the respondents to carefully worded phrases or sentences. Survey instruments (questionnaires) consists of several sections  Survey sections Survey title Section 1: Introductions Explains the reason for the survey Provide assurances that the responses are anonymous Provide details of the time and the place where survey results will be published Provide contact details and ethics approval number  Survey sections Section 2: Instructions Provides instructions to the participants about how to complete the survey For paper-based surveys respondents may not complete all the sections For online surveys, the researcher can ensure that every section must be completed before it progress to the next questions.  Survey sections Section 3: Statements Respondents should rank their answers to the survey statements according to a Likert scale 4,5,7 or a 10 point scale. The 4 point scale and the 10 point scale do not allow the respondents to return a neutral view. Either they agree or disagree The 5 point and 7 point scales do allow for neutral or you decided answer. For some statements it might be appropriate for the respondents to make no comment rather than commit to a view which will bias the result 4 point: Strongly disagree, Disagree Agree, Strongly agree 5 point: Strongly disagree, Disagree, Neutral, Agree, Strongly agree 7 point: Strongly disagree 1 2 3 4 5 6 7 Strongly agree 10 point: Strongly disagree 1 2 3 4 5 6 7 8 9 10 Strongly agree  Survey sections Section 4: Demographics Respondent profiling statements are used to characterize the responses of different sub-groups in the population. Age, height, weight, education, place of birth, years of employment, gender, etc might all be possible reasons for a particular view o be expressed in the survey responses.  Survey sections Example Strongly Statement disagree Disagree Neutral Agree Strongly agree 26-40 My age is < 18 years 18-25 years years 46-65 years 65 years My highest University level of Secondary University higher degree education is Primary school school Apprenticeship degree (M/PhD) I have been employed full time for < 1 year 1-5 years 6-10 years 11–15 years > 15 years  Survey sections Open ended questions and comments The researcher can learn more about the thoughts, background and motivation by inviting responses to open ended questions and comments. Assuming xxxx is the topic of the survey, the open questions and comments can include: List the best attributes of xxxx List the worst attributes of xxxx Give one or more examples of a positive experience using xxxx What needs to be done to make xxx more widely acceptable When reporting the research outcomes it can be useful to include sme exact quotations from insightful comments from a few respondents that support the overall conclusions. It is particularly true in short surveys where the issues cannot be explored in great depth.  Surveytitle Introduction:Outline the reasons to the survey, the survey team, the ethics approval number, research team contact details, timelines, etc. Instructions: Circle/highlight one response in each of the following statements Survey statements Strongly disagree Strongly agree statement 1 1 2 3 4 5 statement 2 1 2 3 4 5 statement 3 1 2 3 4 5 statement 4 1 2 3 4 5 statement 5 1 2 3 4 5 Demographics < 18 18-25 46-65 Myage is (years) years years 26-40 years 65 years University higher Primary Secondary Apprentic University degree My highest level of education is school school eship degree (M/PhD) 6-10 11–15 > 15 I have been employedfull time for (years) < 1 year 1-5 years years years years Open statements 1.What do you consider the best…….  Survey statements Statements should all be positive There is nothing to be gained by making the statements highly complex or varying the statements between positive and negative. Refining you statements Poor statement I don’t feel well most of the time Better statement I feel well most of the time  Survey statements Statements should all be positive There is nothing to be gained by making the statements highly complex or varying the statements between positive and negative. Refining you statements Poor statement I don’t feel well most of the time Better statement I feel well most of the time  Survey statements Statements must be short Surveys must contain concise statements without lengthy explanations. The number of questions must be small. Restrict survey to a maximum of 20 items presented for comment. Surveys are more effective when it can be completed in less than 5 minutes It is unethical to seek more information than what is needed to answer the specific research question. People get frustrated when bombarded with many surveys and this can influence their responses. It is good to inform the prospective respondent how long it will take to complete the survey.  Survey statements Statements must use clear language Statements should not be complex. Use of acronyms, idiomatic phrases and slang must be avoided. Using of double negatives and unusual phrasing must be avoided. The respondents might not understand the statement and give misleading responses. Survey statement examples Hydraulic oil should not be used as a lubricant in electric motors unless Poor statement well insulated. Provided that the motor wiring is well insulated, hydraulic oil is a good Better statement lubricant in electric motors. The EM radiation scattered by an RFID antenna is a measure of Poor statement efficiency. Better statement An efficient antenna will strongly scatter electromagnetic waves The Universitydoes not do a bad job in keeping us informed about Poor statement workplace health and safety issues (double negative) The University does a good job in keeping us informed about workplace  Survey statements Survey statements should not attempt to trick the respondents Survey designers should not see the same information more than once using a number of statements or negative and positive statements to ‘checks the authenticity of responses Trying to trick respondents has a number of negative effects: It can frustrate the participant. This can lead to poor quality feedback from the survey It makes data analysis extremely difficult. If answers are contradictory, which do you choose? If there is a contradiction you may have to discard the survey responses to that statement.  Survey statements Statements should not have multiple themes If two concepts are used in the same statement, then it is impossible for the respondent to choose a single response. Double statements How would you score the following statement? I am satisfied with the ventilation and the lighting environment. If there is a problem with the lighting but not the ventilation, then the respondent cannot convey this information by a single score. A better set of statements is: I am satisfied with the ventilation in the building I am satisfied with the lighting in the building  Survey statements Statements should not direct your feelings onto the respondents It is a relatively simple matter to direct the survey statements in such a manner that the responses that are sought or expected are returned by the participants. The use of emotive words and descriptions must be avoided. Directing the respondent’s feelings using emotive language Poor statement Many people feel ill as soon as they walk into the building. (This statement projects the feelings of the researcher) Better statement People enjoy working in this building.  Survey delivery Paper surveys Can be distributed using mail (post or e-mail) and the respondent is asked to complete the survey and return using the same mechanism –prepaid envelope or scanned pdf document attached to an e-mail. In order to participate the respondents must have a mail or e- mail address. Some section of the populations have neither (either by choice or disadvantage) Researcher may have to travel to a specific location with survey sheet and invite passers-by to answer the questions by oral question and recording the responses on a score sheet. Tone of voice of the questioner can influence the responses.  Survey delivery Respondents with disadvantage A homelessperson or a transient worker is unlikely to have a postal address that is used regularly A large percentage of elderlypeople do not have access to a computer and so will not have an e-mail address People with poor language skills in the language of the survey, nomadic groups and illiterate people cannot be contacted reliably in this manner. It is clearly impossible for a person without a mail address to answer questions about the quality of mail delivery or a person without a car to comment on traffic delays, etc.  Survey delivery Telephone surveys Random selection of contact numbers from a directory, the question are rings numbers and seek responses. The questioner records the responses on a score sheet or computer. The population of respondents is limited to those who can be contacted by publically available numbers at the time of the day when they are available. Online surveys Random selection of e-mail addresses or using social media, the research team can solicit survey responses. The bias in this method lies in the population who regularly use these methods of communication (which since this book was published is probably almost everyone)  Respondent selection Researcher needs to identify the stakeholders. Respondents will participate in a survey for one of more of the following reasons: Public good: Respondent sees the outcome of this research benefiting the community, the nation and the world –no personal benefit, simply seeking to help human kind. Personal good: Respondent sees the outcomes of this research benefiting his/her worldview of how the scientific community and governments should progress the matter under discussion. There might be a long term benefit for him/her later. Personal reward 1: All respondents automatically and anonymously are given a gift –e.g. a redeemable voucher for an online store product or a music download. Personal reward 2: Names of respondents are entered into a prize draw or lottery and a small number of respondents are selected at random to receive this reward.  Respondent selection In the first 3 cases, anonymity can be preserved. Respondent is simply allocated an identification number for inclusion in the spreadsheet. In the last case the name of responded must be included on the document which could compromise anonymity. Ethics approval application must clearly state if and how the respondents will be awarded. Ethics committee needs to approve any incentives offered to participants.  Respondent selection It is important to ensure that one sub-population that dominates the research conclusions. Demographics section of the survey are selected to ensure these different population groups are identified. The differences between two groups of respondents can be assessed statistically using an unpaired t test. If two measurements are done on the same population, the paired t test should be used. The ANOVA analysis should be done for more than two surveys of the same population.  Respondent selection Interested groups of respondents 1. People that live close to proposed new infrastructure developments seek to voice an opinion. 2. Some religious groups seek a voice in new biomedical technologies. 1. The road safety councils have a responsibility to comment on proposed new road safety guidelines and smart highways.  Survey timelines Once the research question has been decided and the research methodology includes a survey (also called the survey instrument)The population of respondents must be determined The instrument wording must be finalized Then ethics approval must be sought Once ethical approval is obtained, the survey must proceed within a quite tightly defined timeline to ensure motivation for participation. Report should not be released before the closing date of the responses. If insufficient responses are obtained it may be necessary to conduct another survey. The larger the number of respondents, the better is the statistical representation of the true population responses. END