NURS 345 Nursing Research - Validity and Reliability (PDF)

NURS 345 NURSING RESEARCH Session 9 Validity and Reliability MS KAFUI HOBENU ([email protected]) Dept. of Research, Education and Administration, SoNM...

NURS 345 NURSING RESEARCH Session 9 Validity and Reliability MS KAFUI HOBENU ([email protected]) Dept. of Research, Education and Administration, SoNM SCHOOL OF NURSING & MIDWIFERY 2024/2025 Academic Year MS KAFUI HOBENU 1  Lecture Overview Reliability and validity are concepts used to evaluate the quality of data collected. They indicate how well a method, technique or test measures something. Reliability and Validity are measures of accuracy in quantitative research whereas the term Research Rigor is used in qualitative research. This session will discuss these concepts that ensures the accuracy of your proposed research SPECIAL  Session Objectives The objectives of this session are to Examine the concepts of reliability and validity Discuss the trustworthiness of qualitative research SPECIAL  Session Outline The key topics to be covered in the session are as follows: Topic one: Validity and Reliability Topic two: Research Rigor SPECIAL Reading List Read Chapter 14 : Polit, D. F., & Beck, C. T. (2014). Essentials of nursing research: Appraising evidence for nursing practice (8th edition.). Philadelphia: Wolters Kluwer Health /Lippincott Williams & Wilkins. Read Chapter 3 Grove, S. K., Burns, N., & Gray, J. R. (2014). Understanding Nursing Research: Building an Evidence-based Practice. Elsevier Health Sciences SPECIAL Validity and reliability Topic One  What is Quantitative Research Quantitative research focuses on gathering numerical data and generalizing it across groups of people or to explain a particular phenomenon Quantitative research, is defined as the systematic investigation of phenomena by gathering quantifiable data and performing statistical, mathematical or computational techniques SPECIAL  What is Quantitative Research Quantitative Research is used to quantify the problem by way of generating numerical data or data that can be transformed into usable statistics. It is used to quantify attitudes, opinions, behaviours, and other defined variables – and generalize results from a larger sample population. Quantitative Research uses measurable data to formulate facts and uncover patterns in research. SPECIAL  Reliability and Validity in Research Reliability and validity are concepts used to evaluate the quality of research. They indicate how well a method, technique or test measures something. ( Middleton, 2019) Reliability is about the consistency of a measure, and validity is about the accuracy of a measure. SPECIAL  Defining Reliability The extent to which results are consistent over time and an accurate representation of the total population under study is referred to as reliability and if the results of a study can be reproduced under a similar methodology, then the research instrument is considered to be reliable. (Joppe, 2000) SPECIAL  Reliability in Quantitative Research Reliability refers to how consistently a method measures something. If the same result can be consistently achieved by using the same methods under the same circumstances, the measurement is considered reliable. SPECIAL  Reliability in Quantitative Research (cont’d) In simple terms, research reliability is the degree to which the research method produces stable and consistent results. A specific measure is considered to be reliable if its application on the same object of measurement a number of times produces the same results. SPECIAL  Examples You measure the temperature of a liquid sample several times under identical conditions. The thermometer displays the same temperature every time, so the results are reliable. A doctor uses a symptom questionnaire to diagnose a patient with a long-term medical condition. Several different doctors use the same questionnaire with the same patient but give different diagnoses. This indicates that the questionnaire has low reliability as a measure of the condition. SPECIAL  Examples Research is reliable if different methods, different researchers and sample groups would produce the same results For example, a medical thermometer is a reliable tool that would measure the correct temperature each time it is used. In the same way, a reliable math test will accurately measure mathematical knowledge for every student who takes it and reliable research findings can be replicated over and over. SPECIAL  Types of Reliability- Characteristics Three types of reliability are referred to in quantitative research, based on these characteristics the degree to which a measurement, given repeatedly, remains the same the stability of a measurement over time; the similarity of measurements within a given time period SPECIAL  Types of Reliability: Test-retest This refers to the consistency of a measure across time, implying that the same results are obtained when one repeats the measurement E.g. A group of participants complete a questionnaire designed to measure attitudes towards contraceptive use. If they repeat the questionnaire days, weeks or months apart and give the same answers, this indicates high test-retest reliability. SPECIAL  Types of Reliability: Inter-rater The consistency of a measure across raters or observers: does one get the same results when different people conduct the same measurement? E.g. Based on an assessment criteria checklist, five examiners submit substantially different results for the same student project. This indicates that the assessment checklist has low inter-rater reliability (for example, because the criteria are too subjective). SPECIAL  Types of Reliability: Internal Consistency The consistency of the measurement itself: Are the same results obtained from different parts of a test that are designed to measure the same thing? You design a questionnaire to measure self-esteem. If you randomly split the results into two halves, there should be a strong correlation between the two sets of results. If the two results are very different, this indicates low internal consistency. SPECIAL  Measuring Reliability Statistically In research, the reliability of an instrument can be measured by the reliability coefficient It is an estimate of the reliability of a measure usually in the form of a correlation coefficient  Commonly used methods for evaluating internal consistency are coefficient alpha or Cronbach’s alpha  Values range from 0.00 to 1.00, the higher the values, the higher the internal consistency reliability SPECIAL  Evaluating internal consistency cont. Coefficient alpha Cronbach’s alpha  Is an index used to A statistic that measures measure the extent to the extent to which all the which the items are items in an instrument reliably measuring the measure the same attribute of interest (Polit attribute (Tavakol & & Beck, 2008) Dennick, 2011) Validity in Quantitative Research Validity refers to how accurately a method measures what it is intended to measure.(Middleton, 2019) Validity is defined as the extent to which a concept is accurately measured in a quantitative study.(Heale &Twycross, 2015) For example, a survey designed to explore depression but which actually measures anxiety would not be considered valid. SPECIAL  Validity in Quantitative Research It is the degree to which an instrument measures what it is supposed to be measured, that is , the accuracy of measurement Validity is concerned with the question: Are we measuring what we think we are measuring? For a measure to be valid, it must be reliable SPECIAL  Validity in Quantitative Research If research has high validity, that means it produces results that correspond to real properties, characteristics, and variations in the physical or social world. E.g. If the thermometer shows different temperatures each time, even though you have carefully controlled conditions to ensure the sample’s temperature stays the same, the thermometer is probably malfunctioning, and therefore its measurements are not valid. SPECIAL  Validity in Quantitative Research High reliability is one indicator that a measurement is valid. If a method is not reliable, it probably isn’t valid. E.g. If a symptom questionnaire results in a reliable diagnosis when answered at different times and with different doctors, this indicates that it has high validity as a measurement of the medical condition. SPECIAL  Validity in Quantitative Research Validity is harder to assess than reliability, but it is even more important. To obtain useful results, the methods you use to collect your data must be valid: the research must be measuring what it claims to measure. This ensures that your discussion of the data and the conclusions you draw are also valid. SPECIAL  Types of Validity: Construct Validity Construct validity evaluates whether a measurement tool really represents the thing we are interested in measuring Construct validity is about ensuring that the method of measurement matches the construct you want to measure To achieve construct validity, you have to ensure that your indicators and measurements are carefully developed based on relevant existing knowledge SPECIAL Construct Validity cont’d If you develop a questionnaire to measure depression, obesity, job satisfaction, intelligence, etc ,you need to know whether the questionnaire really measures the construct of depression, etc? It is therefore determined based on the adherence of a measure to existing theory and knowledge of the concept being measured. SPECIAL  Content Validity Content validity assesses whether a test is representative of all aspects of the construct. The extent to which the measurement covers all aspects of the concept being measured. Eg. When measuring stress levels, it is expected that the questions must cover all aspects of stress by reviewing the work of Hans Selye to determine the areas one needs to question ( alarm, resistance and exhaustion stages) SPECIAL  Content Validity (cont’d) To produce valid results, the content of a test, survey or measurement method must cover all relevant parts of the subject it aims to measure. If some aspects are missing from the measurement (or if irrelevant aspects are included), the validity is threatened. Example: A researcher sets out to measure depression and omits or adds aspects to it that produces results showing anxiety or stress. SPECIAL Face Validity Face validity considers how suitable the content of a test seems to be on the surface. It’s similar to content validity, but face validity is a more informal and subjective assessment. As face validity is a subjective measure, it’s often considered the weakest form of validity. However, it can be useful in the initial stages of developing a method. SPECIAL Face Validity cont’d Example You create a survey to measure the regularity of people’s dietary habits. You review the survey items, which ask questions about every meal of the day and snacks eaten in between for every day of the week. On its surface, the survey seems like a good representation of what you want to test, so you consider it to have high face validity. SPECIAL  Criterion Validity The extent to which the result of a measure corresponds to other valid measures of the same concept. Criterion validity evaluates how closely the results of your test correspond to the results of a different test.(established or widely used test considered valid) To evaluate criterion validity, you calculate the correlation between the results of your measurement and the results of the criterion measurement. If there is a high correlation, this gives a good indication that your test is measuring what it intends to measure. SPECIAL  Reliability Validity What it is The extent to which the results The extent to which the results can be reproduced when the really measure what they are research is repeated under the supposed to measure. same conditions. How it is By checking the consistency of By checking how well the results assessed results across time, across correspond to established theories different observers, and across and other measures of the same parts of the test itself concept. How they A reliable measurement is not A valid measurement is generally relate always valid: the results might reliable: if a test produces accurate be reproducible, but they’re not results, they should be reproducible. necessarily correct. SPECIAL  Assignment Write a one page paper each on the following: Strategies to ensuring Reliability and Validity in research Factors that influence validity and reliability in research SPECIAL Rigor (trustworthiness) in qualitative research Topic Two  Qualitative Research Qualitative research is important because it measures things that numbers might not be able to define. Qualitative methods sometimes identify trends before they show up in the quantitative data. Rigor in qualitative research is the assessment of the trustworthiness. The strength of the research and extent to which one can draw conclusions from it. SPECIAL  Trustworthiness in Qualitative Research The criteria identified by Lincoln and Guba (1986) are credibility transferability dependability Confirmability Since qualitative researchers do not use instruments with established metrics about validity and reliability, it is pertinent to address how qualitative researchers establish that the research study’s findings are credible, transferable, confirmable, and dependable. SPECIAL  Credibility Credibility is how confident the qualitative researcher is in the truth of the research study’s findings. It highlights the truth value of the research ( internal validity). The confidence that can be placed in the truth of the research findings. Credibility establishes whether the research findings represent plausible information drawn from the participants’ original data and is a correct interpretation of the participants’ original views SPECIAL  Strategies to ensure Credibility Strategies to ensure credibility are; Prolonged engagement- long presence during observations, interviews, etc Persistent observation- focusing into detail on issue under study Triangulation- using different data sources Member check – feeding back data to the members SPECIAL  Transferability( generalisability/ external validity) The degree to which the results of qualitative research can be transferred to other contexts or settings with other respondents. Transferability is how the qualitative researcher demonstrates that the research study’s findings are applicable to other contexts. In this case, “other contexts” can mean similar situations, similar populations, and similar phenomena. SPECIAL  Strategies to Ensure Transferability Describing not just the behaviour and experiences, but their context as well, so that the behaviour and experiences become meaningful to an outsider. Transferability concerns the aspect of applicability. Your responsibility as a researcher is to provide a ‘thick description’ of the participants and the research process, to enable the reader to assess whether your findings are transferable to their own setting SPECIAL  Dependability (reliability) Dependability includes the aspect of consistency. You need to check whether the analysis process is in line with the accepted standards for a particular design. Dependability is the extent that the study could be repeated by other researchers and that the findings would be consistent. In other words, if a person wanted to replicate your study, they should have enough information from your research report to do so and obtain similar findings as your study did. SPECIAL  Strategies to ensure Dependability Audit trail- Transparently describing the research steps taken from the start of a research project to the development and reporting of the findings. The records of the research path are kept throughout the study. SPECIAL  Confirmability (Objectivity) Confirmability is the degree of neutrality in the research study’s findings. In other words, this means that the findings are based on participants’ responses and not any potential bias or personal motivations of the researcher The degree to which the findings of the research study could be confirmed by other researchers are not a figment of the researcher’s imagination, but derived from the data. SPECIAL  Strategies to Ensure Confirmability Use of Audit Trail The interpretation should not be based on researcher’s own particular preferences and viewpoints but needs to be grounded in the data. Provide a complete set of notes on decisions made during the research process, research team meetings, reflective thoughts, sampling, research materials adopted, emergence of the findings and information about the data management. etc SPECIAL  Summary Reliability deals with consistency, whereby the researcher ensures that the same results are obtained over time using the same methods Validity deals with the accuracy of the instrument to measure what it is expected to measure In qualitative research, trustworthiness of the research is about measuring its credibility, transferability, confirmability and dependability SPECIAL  References Burns, N., Grove, S. K., & Gray, J. (2015). Understanding nursing research: building an evidence-based practice. 6th edition. St. Louis, Missouri: Elsevier. Polit, D. F., & Beck, C. T. (2010). Essentials of Nursing Research: Appraising Evidence for Nursing Practice: Wolters Kluwer Health/Lippincott Williams & Wilkins. SPECIAL