Critical Eval Study Guide Fall24 PDF

https://quizlet.com/539833740/nyu-critical-eval-study-guide-flash-cards/ Critical Evaluation of Research Study Guide Week 1 What is Evidence-Based Practice (EBP)? Conscientious, explicit, and judicious use of current best evidence in making decisions It is important because of ethics W...

https://quizlet.com/539833740/nyu-critical-eval-study-guide-flash-cards/ Critical Evaluation of Research Study Guide Week 1 What is Evidence-Based Practice (EBP)? Conscientious, explicit, and judicious use of current best evidence in making decisions It is important because of ethics What is the scientific method? Describe the five steps of the scientific method. 1. problem 2. hypothesis 3. procedure 4. analyze 5. Conclusion Describe what it means to say that science is empirical, public, and objective. When we say that science is empirical, we mean that answers are obtained by making observations. Although preliminary answers or hypotheses may be obtained by other means, science requires empirical verification. The scientific method is public. By this we mean that the scientific method makes observations available for evaluation by others, especially other scientists. The scientific method is objective. That is, the observations are structured so that the researcher's biases and beliefs do not influence the outcome of the study.​ Empirical: making observations to get answers Public: submitting information for critique by others Objective: leaving our opinions or bias aside Define a hypothesis. An idea or explanation based on limited evidence to make a starting point for a possible study or experiment What do I think is going to happen based off my basic information What are the four elements that need to be included in foreground questions? PICO Patient, population or problem Intervention, treatment Comparison; what are we comparing it to Outcome; desired change For each of the following, identify the section of a research paper that would probably contain the desired information: o How many individuals participated in the study, and what are their characteristics? Methods o Why was the study done? Introduction o Did the study produce a statistically significant result? Results o What are the limitations of the study and directions for future research? Discussion Week 2 What is the Nuremberg Code? Was created post WWII after the atrocities. Doctors were experimenting on people without consent and using extreme procedures. Make sure to confirm consent, safety, qualification, benefits. What are four major ethical concerns/issues with research? No harm, Obtain informed consent, Use of deception and debriefing, Confidentiality and anonymity; slide 18 Name several places where you can search for peer-reviewed articles: NYU Libraries website, google scholar, ASHA, pubmed Nuremberg Code (voluntary consent of human subject is essential; avoid harm and risk) Milgrim Study of authority (teacher/learner and shocks) Monster Study (stuttering children) Stanford Prison Study (guards/prisoners; effect of authority) Week 3 Define a construct. A broad idea or concept or topic that may or may not be observable (observable) Define a variable. Characteristics or conditions that vary ex. V1 V2 (stuttering vs. not stuttering) Some variables are inherently well defined and easily measured, whereas some variables are abstract. Identify examples of each. Easily measured: height and weight Abstract: intelligence, self esteem  Briefly explain what an operational definition is and why operational definitions are sometimes necessary. A procedure for measuring and defining a construct/variable (abstract --> concrete) Specifies a measurement procedure for measuring an external, observable behavior How would you observe and write a definition or description; non standardized How may you operationally define the following? o Receptive vocabulary: pointing to something, circling the answer o Expressive vocabulary: narratively saying something “I see the brown bear under the tree” o Hoarseness: measure of quality of voice o Breathiness: how much air, having a naive listener listen to the breathiness Define the validity of measurement. The degree to which a tool of measurement accurately measures what it said it would measure; does it do what it said it would do Define the reliability of measurement. The degree to which the same results happen as a result of the same conditions and treatment Describe the relationship between validity and reliability as they pertain to measures. A measure cannot be valid if it isn't reliable, but not the other way around. Bathroom scale analogy; a bathroom scale that says 5 lbs every time is reliable because it has the same result but it is not valid because it doesn’t measure what it said it would measure Types of validity o concurrent is demonstrated when scores obtained from a new measure are directly related to scores obtained from an established measure of the same variable. Correlations: positive, negative, none (examine scatterplots) o predictive is demonstrated when scores obtained from a measure accurately predict behavior according to a theory. o Construct requires that the scores obtained from a measurement procedure behave exactly the same as the variable itself. This type of validity is based on many research studies that use the same measurement procedure and grows gradually as each new study contributes more evidence. Idea or a goal Define test-retest reliability and inter-rater reliability. the consistency of scores when administered on multiple occasions; admisteding the same test; multiple occasion one observer Different tests measuring the same thing; agreement between multiple tests; same occasion, multiple observers A speech-language pathologist has developed a new test for measuring receptive language and would like to determine the validity of the test. The new test and an established measure of receptive language are both administered to a sample of participants. Describe the pattern of results that would establish concurrent validity for the new test. The pattern of results could be positive, negative or none on a scatter plot. Looking for a positive trend correlation between this study vs other studies of the same experiment What are the four scales of measurement? Identify examples of each. Nominal: qualitative; gender, Ordinal: sequentially; 1-5, 1-10, education level Interval: temperature Ratio; number of stutters, number of r in arctic Compare and contrast interval and ratio scales. Interval has an arbitrary zero and ratio has a true zero. You can get zero right on an arctic screener. The same because they are both sequential and the categories are the same size. What is a ceiling effect, and how can it be a problem? When everyone had a positive result; everyone got 100 on the math test Was it challenging enough, was there something wrong. Everyone scored high. What are demand characteristics, and how do they limit the validity of the measurements obtained in a research study? Demand characteristics lead participants to know what is being studied, so participants can act a certain way or answer a certain way that they think the researchers want. This would affect external validity Reactivity: they came back and answered a certain way What is a population? What is a sample? Describe the relationship between a population and a sample. a whole. Its every member of a group, the larger group of interest What is a sample? a fraction of the population, the smaller subset that is worked with Sample is taken from the population The sample should be representative of the population so that the researchers are able to generalize findings. The sample should have different ages, severities, races, genders Under what circumstances can a sample be biased? If a sample is not representative it can be biased. We identified the five research designs (descriptive, correlational, experimental, quasi-experimental, and nonexperimental). We can group these five research designs into three broad categories based on the question being addressed and data structure: Experimental, differential and correlational 3 groups o Designs that examine individual variables (instead of relationships between variables) descriptive o Designs that examine relationships between variables by measuring two (or more) variables for each participant correlational o Designs that examine relationships between variables by comparing two (or more) groups/sets of scores. Experimental, quasi or non experimental Describe the concept of a study’s internal validity. What are the common threats to internal validity? internal validity is established if there is a single unambiguous explanation for the relationship between the variables of interest What are the common threats to internal validity? Environmental variables: time of day, room, experimenter characteristics, temperature, amount of light... Individual differences: char's that differ from one person to another Time-related variables: One explanation for a result Describe the concept of a study’s external validity. What are the common threats to external validity? generalizability of the results to people,settings, times, measures, and characteristics other than those in the study ○ Experimental research studies tend to have very (high) internal validity but often have relatively (low) external validity. Because they did a good job and made no nuisance variables it is very specific and hard to generalize Week 4 Define independent, dependent, and extraneous variables. I: the variable that is manipulated by researcher and consists of two or more treatment conditions D: the variable that is observed for changes to assess effects of manipulation of the independent variable E: all other variables What are the defining characteristics of a between-subjects design as compared to a within-subjects design? Between: One treatment for each person: Compares scores from different groups of individuals Therefore, scores are considered independent because scores are provided by separate, unique participants, In an experiment, each group is assigned to one treatment condition Within: Compares scores in two or more different treatment conditions by observing or measuring the same individuals in each condition, We look for differences between treatment conditions by looking at changes within the same group of participants, Each individual in the study has more than one score Explain how the following factors can threaten the internal validity of a between-subjects design. o Individual differences between groups: produce high variability in scores within groups o Differential attrition: if participants who share common characteristics (or those from the same group) leave the study o Communication between groups: diffusion is the spread of treatment effects from the experimental group to the control group (think about if a drug group tells the placebo group how well they're doing...) What are the three primary techniques for limiting confounding by individual differences in between-subjects experiments? Random assignment, Matching groups to balance occurrence of a variable, Hold a variable constant; increase sample size Explain the general advantages and disadvantages of a within-subjects design. Advantages: Requires fewer participants than between-subjects (because each participant is in each condition), eliminates/reduces all problems based on individual differences as confound, Disadvantages: Participant performance is susceptible to time-related factors, threatening internal validity, Fatigue, Environmental changes between conditions, Practice effects, Participant attrition (greater problem with longer intervals between treatment conditions. Describe how factors such as testing effects, history, maturation, and instrumentation as well as statistical regression can threaten the internal validity of a within-subjects design. Because participant performance is susceptible to time-related factors, it threatens internal validity. Exhaustion, time related can create a low internal validity because there are lots of nuisance factors What are the three measures of central tendency? Mean, median, mode What are the two measures of variability? Standard deviation and variance Define mean and standard deviation. Mean is the average, SD is the average distance from the mean, ex. How many SD from the mean Explain how the purpose of the descriptive research design differs from the purposes of experimental or nonexperimental research designs. With descriptive research, we present descriptive statistics, we are not looking at relationships between variables, just at the variables themselves. Involved measuring a variable or set of variables as they exist naturally Describes current state What is a correlation coefficient? (Hint: What does it describe?) Describes the direction of the relationship, Describes the form of the relationship (usually), Linear or monotonic, The consistency of the relationship A correlation coefficient describes the direction of the relationship between the two variables. In a positive relationship, there is a tendency for two variables to change in the same direction; as one variable increases, the other tends to increase. In a negative relationship, there is a tendency for two variables to change in opposite directions; an increase in one variable tends to be accompanied by a decrease in the other. Week 5 What are the four basic elements that an experimental design contains? Manipulation, Measurement, Comparison, Control  In nonexperimental and quasi-experimental designs, how are the groups or conditions defined? How does it differ from an experimental design? Researcher does not create groups, so vulnerable to assignment bias. Quasi no randomization, experimental yes randomization. Researcher creates groups or conditions, assignment bias can be accounted for by matching, random assignment, control (True) Nonexperimental and quasi-experimental studies always contain a threat to internal validity. There are two types of developmental research designs. Cross sectional design evaluates changes in behavior related to age by examining different groups of individuals with each group representing a different age. longitudinal design evaluates changes in behavior related to age by examining one group of participants who are roughly the same age, at different times. Identify three problems in establishing causation. In natural situations, changes in one variable are accompanied by changes in many variables E.g., increased food intake à increasing weight The third variable problem, changes in one variable are accompanied by changes in many variables, The directionality problem: even if you can establish a relationship between two variables, determining which is the cause and which is the effect can be difficult, Controlling nature: in order to establish causation, we must employ strict controls Researchers indicate that people who suffer from depression also tend to experience insomnia. However, it is unclear whether the depression causes insomnia or the lack of sleep causes depression. What problem is demonstrated by this example? (Select one from the following) a. the directionality problem b. the third-variable problem c. the extraneous variable problem Explain how extraneous variables can become confounding variables. they can influence the independent variable or the dependent variable and can cause a third variable problem Environmental Individual differences Time-related variables Identify the three methods of actively controlling for extraneous variables. Holding a variable constant, Matching across treatment conditions, Control by randomization (random assignment) Holding a variable constant is a technique for removing one threat to (internal validity), but it can limit the (external validity) of an experiment. The Zimbardo prison study is an extreme example of a timulation experiment. Researchers often use simulation experiments in an attempt to increase the (external) validity of the results. Trying to generalize Field studies tend to have (lower) internal validity and (higher) external validity than experimental research studies. Nonexperimental and correlational designs tend to have less control than the experimental design. How does this affect the internal and external validity of nonexperimental and correlation studies? (This question is related to topics from week 4 (e.g., a balancing act between internal and external validity). Hint: Look at the figure below.) Nonexperimental and correlational studies tend to have lower internal validity and higher external validity. Because they have less control compared to experimental where everything is controlled. Week 6 Define a factorial design, including the term factor. a design that includes two or more independent variables Factor: independent (predictor) variables A 2 X 3 factorial design is a two-factor design with 2 levels of the first factor and 3 levels of the second factor.  Define a main effect for one factor and an interaction between factors. the mean differences among the levels of one factor Occur when two factors acting together produce mean differences that are not explained by the main effect of the two factors, Occur when the effect of one factor depends on the different levels (or value) of a second factor Differences in mean scores on BNT for Tx A differ from Tx B depending on whether a person received treatment 1x/week or 5x/week Refer to slide 12 and 13 for visual A factorial study measures allergy symptoms before and after taking medication for a group taking the real medication and a control group taking a placebo. What kind of design is being used? a. Between-subjects design b. Within-subjects design c. Repeated measures design d. Mixed design Describe the two general categories of statistics, descriptive and inferential, and explain the purposes of each. (Refer to week 4 materials to identify the goal for descriptive statistics.) Two categories of statistics are descriptive and inferential Descriptive statistics: Depending on the level of measurement of data, different numbers are used to describe that data Purpose of descriptive statistics: Provide basic information about variables in the dataset and highlight potential relationships between variables Inferential: Used when we are trying to infer characteristics about a population from a small sample of participants Purpose of inferential statistics: To use limited sample information as the basis for reaching general conclusions about the population Define the null hypothesis and the alternative hypothesis. a statement about the population indicating that there is no effect, no change, no difference, no relationship the research hypothesis that there is an effect/change/difference/relationship Explain what p <.05 means. there is less than a 5% chance that our data resulted from chance...meaningthat there's a 95% chance that our results were due to the variables that we were studying and not just due to random chance. We can a low p; we want it to be close to 1 Define effect size. a standardized measure of the magnitude of an observed effect Cohen’s d, Pearson’s correlation coefficient r... r =.1 (small effect)—explains 1% of total variance r =.3 (medium effect)—explains 9% of total variance r =.5 (large effect)—explains 25% of total variance What elements are required for a single-case research study to qualify as an “experiment”? MUST include manipulation of IV, MUST have strict control of extraneous variables Single-subject/single-case designs (experimental) Use results from one participant to establish cause-effect relationship between variables Only one participant (sometimes more than one), so can’t use certain statistical tests to evaluate results (e.g., t-tests) Instead, we visually inspect graphs of participant’s data see slide 40 Traditional statistics (means, variances, and hypothesis tests) are not used to evaluate the results from a single-case study. Explain how the results are evaluated. Visual inspection of a graph is how we decide whether changes are substantial enough to be considered a large enough effect. What name is given to a single-case design consisting of the following four phases in the order given: baseline, treatment, baseline, treatment? ABAB What is an ethical concern for the ABAB design? It isn't ethical to completely withdraw treatment, especially if the treatment effects are maintained. What happens if we aren't able to get these good results again?If someone pulled out we lose that data Describe the structure of a multiple-baseline design. Series of A-B designs stacked on top of one another, with each baseline phase progressively longer Collect baseline data, then apply treatment in a staggered fashion to multiple levels of a target variable Different behaviors (e.g., target speech sounds, morphemes, grammatical structures) Different participants Different settings Frequently used in CSD because not necessary to demonstrate that gains are eliminated after withdrawal of treatment Two simultaneous baselines for two different b/h's or two different participants (baseline observation extends for one b/h or participant for different period of time) THEN treatment applied for one b/h or participant; baseline continues for other b/h or participant, then applied later. Identify the primary strength of a multiple-baseline design. The researchers dont have to remove/withdraw the treatment after it has been shown to be effective Frequently used in CSD because not necessary to demonstrate that gains are eliminated after withdrawal of treatment; we have already given the speech therapy rather than giving a pill and then taking it away to see how it changes Statistical tests (Week 4 & Week 6) Identify the appropriate statistical test for each of the following situations: she said to know this o A between-subjects study comparing two groups’ means Independent-samples t-test o A between-subjects study comparing means of more than two groups analysis of variance (ANOVA) o A within-subjects study comparing two groups’ means dependent samples t test With nominal dependent/outcome variables Compare proportions of individuals in each category: chi-square Test With ordinal-level dependent/outcome variable  Comparing two groups’ medians: Mann-Whitney’s U Comparing more than two groups’ medians: Kruskal-Wallis test Week 7 What differentiates systematic reviews and meta-analyses from traditional literature reviews? In a systematic review: we are just reviewing Standard procedure for literature search (searching for papers about a topic), Operationalized criteria for including/excluding studies from review (specific and intentional, why they chose what they chose), multiple reviewers independently evaluate sources (we want to see reliability (same)) Helpful: because you can look at one review rather than multiple experiments Meta-analysis: we are doing math and comparing; take numbers to calculate the magnitude of these findings a type of systematic review that generates one or more quantitative summary statistics Calculate average effect size across studies May be weighted for sample size May calculate 95% confidence interval May not be possible to make these calculations if adequate information not provided in original Studies (not all studies provide enough information to adequately make a good meta analysis) Traditional lit review decisions are made by the author about what studies to review and what findings are relevant to the study that was conducted Introduction section of the research Can be biased because its one viewpoint they wouldn’t want views that are different Describe a primary difference between systematic reviews and meta-analyses. Describe a primary difference between systematic reviews and meta-analyses: The results of a meta-analysis are presented quantitatively in the form of a summary statistic A key difference between a systematic review and a meta-analysis is that a systematic review summarizes all available evidence on a topic by critically appraising studies, while a meta-analysis takes the results from those studies and statistically combines them to produce a quantitative estimate of the overall effect. Review: we are just reviewing and looking at stuff, meta we are doing the math and comparing. How much weight should a patient’s preference carry in clinical decision making in the context of EBP? The ethical principles of autonomy and beneficence should supercede -> preference of a fully informed patient should be of primary importance: Most important thing Always honor your patient/client’s choice “among efficacious programs...unless there is compelling evidence that one approach is substantially superior to others