W15-L12 (DSA 458) Questionnaire Development PDF

Research Methodology (DSA 4580) Step 3: Designing a Study and Collecting Data Questionnaire Development Academic Year: 2023-2024 Fifth Year DSA 4580 WEEK 15 LECTURE 12 Abdulhamid Al Ghwainem, BDS, MSc, DClinDent Pediatric Dentistry Assistant Professor in Pediatric Dentistry [email protected]....

Research Methodology (DSA 4580) Step 3: Designing a Study and Collecting Data Questionnaire Development Academic Year: 2023-2024 Fifth Year DSA 4580 WEEK 15 LECTURE 12 Abdulhamid Al Ghwainem, BDS, MSc, DClinDent Pediatric Dentistry Assistant Professor in Pediatric Dentistry [email protected] Thursday, 30 November 2023 Copyright © 2023 by PSAU, Abdulhamid Al Ghwainem Disclaimer DISCLAIMER The information presented in this lecture is offered for educational and informational purposes and should not be construed as medical, dental, or research advice. While the amount of information in this handout is vast, and I make every effort to be as current and thorough as possible, the information cannot be taken as a reference manual or textbook. Please note that you should read the required textbooks as specified in the course curriculum and lecture references. 2 Notice WARNING Materials used in connection with this course or lecture may be subject to copyright protection. This material has been reproduced and communicated to you by or on behalf of Prince Sattam Bin Abdulaziz University. Materials may include, but are not limited to: documents, slides, images, audio, and video. Materials in this course are only for the use of students enrolled in this course, for purposes associated with this course, and may not be retained for longer than the class term. Unauthorized retention, duplication, distribution, or modification of copyrighted materials may be the subject of copyright protection by law. For more information, visit the Saudi Authority for Intellectual Property. Do not remove this notice 3 Outline § Overview of the questionnaire in scientific research. § Differences between survey and questionnaire. § The questionnaire design and development in terms of administration, question types, wording, order, formatting, and layout. § The questionnaire's validity and reliability. § Practice, discussion and example of selfadministered questionnaire. 4 Lecture Objectives By the end of this session, you should be able to: • Define survey research. • Differentiate between a survey and a questionnaire. • Identify the various ways to administer a survey and understand the limitations of each survey method of administration. • Describe the characteristics of practical survey questions. • Identify the multiple aspects of a survey instrument's validity and reliability. • Recognize some of the available online survey tools that can be used in research projects. 5 Research Process/Steps Research: answering questions in logical and systematic ways Question Answer Research methodology: how to get from Question to Answer 6 Research Process/Steps Research: answering questions in logical and systematic ways Question Answer Research methodology: how to get from Question to Answer Identify study question Select study approach Design study and collect data 7 Analyse data Report findings Research Process/Steps Research: answering questions in logical and systematic ways Identify study question Select study approach Design study and collect data • Research Ethics • Sampling Techniques • Questionnaire Design 8 Analyse data Report findings Questionnaire Design: Overview § What is a questionnaire? What is a survey? Are they the same? What are the differences? Why, when, and how to use a questionnaire or survey?. This Photo by Unknown Author is licensed under CC BY-SA 9 Questionnaire Design: Overview § A questionnaire (or survey instrument/tool) is a series of questions used as a tool for systematically gathering data from study participants. This Photo by Unknown Author is licensed under CC BY-SA 10 Questionnaire Design: Overview § It consists of a list of questions or items used to gather data from respondents about their knowledge, attitudes, experiences, or opinions. § Questionnaires can be used to collect quantitative and/or qualitative information. § Questionnaires are commonly used in scientific research, particularly in health research. § For example, dental researchers may investigate patients' experiences about clear aligners using questionnaires. § Can you think of another example? 11 Questionnaires vs. surveys § Surveys and questionnaires are the two methods of acquiring primary data from the respondents. § ‘Survey’ is an umbrella term that includes a questionnaire, interview, and observation method as a tool for collecting information. § Surveys are the conventional way of conducting research in which the respondents are questioned regarding their behavior, awareness, motivations, demographics, and other characteristics. § Conversely, Questionnaires are a tool for acquiring data on a particular topic, which involves distributing forms that comprise questions relating to the topic under study. § The two terms are often used interchangeably, but there are some differences. 12 Questionnaires vs. surveys § The best, quickest, and inexpensive way of conducting a survey is the questionnaire. § Both surveys and questionnaires use a series of questions to gather information, however, the purpose of the research and the handling of the data, after it is collected distinguish a questionnaire from a survey, e.g.: § A questionnaire is a set of written questions used for collecting information for the benefit of one single individual. § A survey is a process of gathering information for statistical analysis to benefit a group of individuals (a research method). 13 Questionnaires vs. surveys Survey Questionnaire Definition Survey refers to the collection, recording and analysis of information on a particular subject, an area or a group of people. Questionnaire implies a form containing a list of ready-made questions, delivered to people for obtaining statistical information. What is it? Process of collecting and analysing data Instrument of data collection Use It is conducted on the target population. It is distributed or delivered to the respondents. Time Time consuming process Fast process Open/close ended Closed ended Subjective or Objective Objective Questions Answers Adapted from Surbhi, S. (2016). Difference between survey and questionnaire. Retrieved from https://keydifferences.com/difference-between-survey-and-questionnaire.html 14 Questionnaire Design: § The first step in designing a questionnaire is to list the topics that the survey instrument must cover and decide how to deliver it. § The first set of questionnaire items typically enables the researcher to confirm that participants meet the eligibility criteria for the study. § Several questions may be required in order to accurately assign participants to key exposure and disease categories. 15 Questionnaire Design: 16 Questionnaire Design: Administration § Self-administered: § Self-administered questionnaires can be delivered online or in paper-and-pen formats, in person or through mail. All questions are standardized so that all respondents receive the same questions with identical wording. Advantages: Disadvantages: § Cost-effective. § Unsuitable for people with limited literacy or verbal skills. § Easy to administer for small and large groups. § Susceptible to a nonresponse bias (most people invited may not complete the questionnaire). § Anonymous and suitable for sensitive topics. § Self-paced. § Biased towards people who volunteer because impersonal survey requests often go ignored. 17 Questionnaire Design: Administration § Researcher-administered: § Researcher-administered questionnaires are interviews that take place by phone, in-person, or online between researchers and respondents. Advantages: Disadvantages: § Help to ensure that the respondents are representative of the target population. § Costly and time-consuming to perform. § Allow clarifications of ambiguous or unclear questions and answers. § Have high response rates because it’s harder to refuse an interview when personal attention is given to respondents. 18 § More difficult to analyze if you have qualitative responses. § Likely to contain experimenter bias or demand characteristics. § Likely to encourage social desirability bias in responses because of a lack of anonymity. Questionnaire Design: Administration 19 Questionnaire Design: Administration 20 Questionnaire Design: Types of Questions § The next step is to decide which types of questions are most appropriate. § Closed-ended questions allow a limited number of possible responses. § Open-ended questions (also called free-response questions) allow unlimited possible responses. 21 Questionnaire Design: 22 Questionnaire Design: Types of Questions Closed-ended questions allow a limited number of possible responses. § Closed-ended or restricted-choice questions offer respondents a fixed set of choices from which to select. § Closed-ended questions are best for collecting data on categorical or quantitative variables. § Categorical variables can be nominal or ordinal. § Quantitative variables can be intervals or ratios. § Understanding the variable type and measurement level means you can perform appropriate statistical analyses for generalizable results. 23 Questionnaire Design: Types of Questions Examples of Closed-ended questions: Nominal Variables: include categories that can’t be ranked, such as race or ethnicity. § This includes binary or dichotomous categories. § It’s best to include categories that cover all possible answers and are mutually exclusive. § There should be no overlap between response items. § In binary or dichotomous questions, the respondents only have two options to choose from. 24 Questionnaire Design: Types of Questions Examples of Closed-ended questions: Nominal Variables: include categories that can’t be ranked, such as race or ethnicity. What is your race? q Arab q Middle eastern q White q Black or African-American q Asian Are you aware of the Child Protection Law in Saudi Arabia? q Yes q No q I don’t know What is your race? q Arab q White q Black or African-American q Asian Are you aware of the Child Protection Law in Saudi Arabia? q Yes q No 25 Questionnaire Design: Types of Questions Examples of Closed-ended questions: Ordinal Variables: include categories that can be ranked, such as age and rating scales. How old are you? q 18 or younger q 18-30 q 30-40 q 40-50 q 50 or older LIKERT SCALE questions collect ordinal data using rating scales with 5 or 7 points How satisfied or dissatisfied are you with your Research Methodology course? q Very dissatisfied q Somewhat dissatisfied q Neither satisfied nor dissatisfied q Somewhat satisfied q Very satisfied How old are you? q 18 or younger q 19-29 q 30-39 q 40-49 q 50 or older 26 Questionnaire Design: Types of Questions Closed-ended questions come in a variety of formats: § Date and time questions. § Numeric questions. § Categorical. –Ranked (ordinal) categorical answers have an inherent order –Unordered (nominal) categorical answers do not have any built-in order § Paired comparisons. § Rank-ordering questions. 27 Questionnaire Design: Types of Questions 28 Questionnaire Design: Types of Questions § For closed-ended questionnaire items, the researcher must decide what types of response options are appropriate. –The responses should be ones that participants can record accurately and completely. § For numeric responses, the question should state exactly how specific the answers should be. § For categorical questions, researchers should consider all possible responses and determine how many response options are needed to allow all participants to answer each question. § For ranked questions, the researcher chooses how many entries to include on the scale and whether there will be a neutral option. 29 Questionnaire Design: Question Wording and Phrasing § Question wording can influence the respondents’ answers, especially if the language is unclear, ambiguous, or biased. § Good questions need to be understood by all respondents in the same way (reliable) and measure exactly what you’re interested in (valid). § Question wording and phrasing should be clear, focused, and balanced. § Avoid jargon, abbreviations, vague and ambiguous language, fault assumptions, sensitive questions, etc. 30 Questionnaire Design: Question Wording and Phrasing 31 Questionnaire Design: Question Order § The questionnaires can be ordered logically, with a clear progression from simple to complex. § Alternatively, the question order between respondents can be randomized. § Many questionnaires start with easy or at least general questions before moving to more difficult. § It is often best to group similar questions with similar response types, so that they are asked consecutively. – Sometimes it is better to mix up such questions to prevent habituation, an error that occurs when participants become so accustomed to giving a particular response (like “agree… agree… agree…”) that they continue to reply with the same response even when that does not match their true perspectives. § Survey developers must carefully consider how previous questions could taint the answers to later ones. 32 Questionnaire Design: Layout and Formatting Both paper-based and electronic data entry forms require careful attention to layout and spacing. § Paper-based and electronic survey pages should be carefully checked for grammatical errors, misspellings, missing questions, gaps in logic, unclear instructions, formatting errors, and other organizational issues. § The selected typefaces and fonts must be readable and sufficiently large. 33 Questionnaire Design: Layout and Formatting § Computer-based data entry programs allow a researcher to force participants to provide an answer to selected questions, but required fields force a person who does not want to provide an answer to one required question to quit the survey and leave all subsequent items unanswered. § Skip logic codes can automatically hide irrelevant questions from participants based on their responses to filter questions. 34 Questionnaire Design: Translation § Back translation (or double translation): one person translates the questionnaire from the original language to a new language, and then a second person translates the survey instrument in the new language back into the original language. § A comparison of the original version of the survey with the back-translated version will reveal where the second-language translation does not match the intended meaning of the original version. 35 Questionnaire Design: Reliability and Validity § A reliable and valid questionnaire (or other assessment tool) measures what it was intended to measure in the population being assessed. § Reliability, or precision, is demonstrated when consistent answers are given to similar questions and when an assessment yields the same outcome when repeated several times. § The validity, or accuracy, of a survey instrument (or a diagnostic test or other assessment tool) is established when the responses or measurements are shown to be correct. 36 Questionnaire Design: Reliability and Validity 37 Questionnaire Design: Reliability and Validity § Internal consistency is present when the items in a survey instrument measure various aspects of the same concept. § Internal consistency in a data set can be confirmed with tests of intercorrelation such as Cronbach’s alpha and the KR-21 –Cronbach’s alpha is used with variables that have ordered responses –Kuder-Richardson Formula 21 (KR-21) is used with binary variables 38 Questionnaire Design: Reliability and Validity § Test-retest reliability is demonstrated when people who take a baseline assessment and then re-take the test later have about the same scores each time they are tested. § Inter-observer agreement or inter-rater agreement describes the degree of concordance among independent raters assessing the same study participants. –The kappa statistic (Cohen’s kappa) determines whether two assessors who are evaluating the same study participants agreed more often than is expected by chance. 39 Questionnaire Design: Reliability and Validity 40 Questionnaire Design: Reliability and Validity § Content validity (logical validity): subject matter experts agree that a set of survey items captures the most relevant information about the study domain. –Face validity: content experts and users agree that a survey instrument will be easy for study participants to understand and correctly complete. 41 Questionnaire Design: Reliability and Validity § Some statistical methods can provide information about which items in an assessment tool might be redundant or unnecessary so they can be removed without compromising the validity of the survey instrument. § Principal component analysis (PCA) creates one or more index variables (called components) from a larger set of measured variables. 42 Questionnaire Design: Reliability and Validity § Construct validity: a set of questions in a survey instrument measures the theoretical construct the tool is intended to assess. –Factor analysis uses measured variables to model a latent variable that represents a construct that cannot be directly measured with one question but appears to have a causal relationship with a set of measured variables. § Convergent validity: two items that the underlying theory says should be related are shown to be correlated. § Discriminant validity: two items that the construct says should not be related are shown not to be associated. 43 Questionnaire Design: Reliability and Validity § Criterion validity (concrete validity) uses an established test as a standard (or criterion) for confirming the utility of a new test that examines a similar theoretical construct. –Concurrent validity is evaluated when participants in a pilot study complete both the existing and new tests, and the correlation between the test results is calculated. –Predictive validity is appraised when the new test is correlated with subsequent measures of performance in related domains. 44 Questionnaire Design: Online Survey Tools § Several widely used questionnaires and survey tools are available to researchers. § Some of these tools are free of charge, but most are commercial products that require payment for use. 45 Questionnaire Design: Pilot Testing § A pilot test (or pretest) is a small-scale preliminary study conducted to evaluate the feasibility of a full-scale research project. § This is to make sure what you’re asking is always clear and unambiguous. § Pretesting helps to find any errors or confusion before performing your study. § This can be done by distributing the questionnaire to a small number of friends, classmates, or members of your target population to complete your questionnaire using the same method that will be used for the research. § Find out if any questions were particularly difficult to answer or if the directions were unclear or inconsistent, and make changes as necessary. § The survey instrument is then revised based on these observations. 46 Break Out Session Instructions: Scan the QR and complete the questionnaire 47 Break Out Session Feedback 48 Break Out Session Feedback § The survey tool (questionnaire) was constructed based on previous studies (Adair, 1997; Ramos-Gomez et al., 1998; John et al., 1999; Kilpatrick et al., 1999; Cairns et al., 2005a; Thomas et al., 2006a; Al-Habsi et al., 2009; Chadwick et al., 2009; Harris et al., 2009b; Harris et al., 2009a; Owais et al., 2009; Sonbol et al., 2012; Hashim and Al-Ani, 2013; Al-Dabaan et al., 2014; Duman et al., 2021; Sulimany et al., 2021). § Two experts assessed the questionnaire's language, phrasing, and comprehensiveness. Minor adjustments to the questionnaire were made based on the experts' comments. § This is a structured, self-reported questionnaire written in English. § Consisted of 30 items across four sections. § The first section was to survey the demographic information and educational background, including age, gender, nationality, affiliated dental school, educational background, and information regarding previous CAN teaching. § The second section consisted of nine items related to the assessment of CAN knowledge, including identifying CAN types, risk factors for CAN, manifestations of physical abuse, and indicators of CAN. § The questionnaire's third section contained seven items related to the respondents’ attitudes toward CAN. § The last section included five items designed to survey the respondents’ practice toward CAN in their dental schools, actions taken for suspected cases, and whether their dental schools have a protocol for dealing with CAN-suspected cases. § The response formats utilized for these sections encompassed binary responses with yes/no answers, categorical responses in multiple-choice format, or selecting a response based on a five-point Likert scale. 49 Break Out Session § § § § The questionnaire was piloted to assess the content and face validity by conducting a pilot survey of final-year dental students at Prince Sattam Bin Abdulaziz University (n=30). Moreover, the questionnaire's validity and reliability were assessed by asking final-year dental students to complete the questionnaire twice in one week between each completion. The overall validity and reliability were calculated using SPSS statistical package version 26 (IBM Corp., Armonk, NY, USA) and were reflected by an average Cronbach's alpha of 0.822, kappa (k) of 0.669 and Intraclass Correlation Coefficient (Cicchetti and Sparrow) of 0.883 (Tab. 1, Tab. 2, and Tab. 3), which is deemed acceptable to a good agreement. The validation and reliability processes made minor changes to the format, structure of responses, and mandatory nature of specific questions. Feedback 50 References Required: Neale, J., 2020. Research methods for health and social care. Bloomsbury Publishing. Jacobsen, K.H., 2020. Introduction to health research methods: A practical guide. Jones & Bartlett Publishers. Additional: Bowling, A., 2014. Research methods in health: Investigating health and health services. Maidenhead, United Kingdom: Open University Press. Creswell, J.W. and Creswell, J.D., 2018. Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications. World Health Organization. Regional Office for the Western Pacific, 2001. Health research methodology : a guide for training in research methods. 2nd ed.. WHO Regional Office for the Western Pacific. https://apps.who.int/iris/handle/10665/206929 World Health Organization. Regional Office for the Eastern Mediterranean. (2004). A practical guide for health researchers. https://apps.who.int/iris/handle/10665/119703 References Cited references: § Surbhi, S. (2016). Difference between survey and questionnaire. Retrieved from https://keydifferences.com/differencebetween-survey-and-questionnaire.html Next Lecture: Step 3: Designing a Study and Collecting Data Research Proposal Identify study question Select study approach Design study and collect data Thank you! Any questions [email protected] Analyse data Report findings