Chapter 3: Defining and Measuring Variables - Methodology PDF

methodologie Chapter 3: defining and measuring variables =========================================== 3.1 constructs and operational definitions ========================================== The research process: Find an unanswered question that will serve as your research idea\ =\> form a hypothesi...

methodologie Chapter 3: defining and measuring variables =========================================== 3.1 constructs and operational definitions ========================================== The research process: Find an unanswered question that will serve as your research idea\ =\> form a hypothesis\ =\> use hypothesis to develop an empirical research study that will support or refute the hypothesis\ =\> define how each of the variables will be measured [Theories and constructs] - Theories that contain hypothetical mechanisms and intangible elements are developed to explain and predict behaviour - (hypothetical) constructs are hypothetical attributes or mechanisms that help explain and predict behaviour in a theory\ =\> can be influenced by external stimuli and, in turn, can influence external behaviours [Operational definitions] = a procedure for indirectly measuring and defining a variable that can't be observed or measured directly, it specifies a measurement procedure (set of operations) for measuring an external, observable behaviour and uses the resulting measurements as a definition and a measurement of the hypothetical construct Can also be used to define variables to be manipulated [Limitations of operational definitions] - It's not the same as the construct itself\ =\> concerns about quality of operational definitions and the measurement they produce - There's no one-to-one relationship between the variable that's being measured and the actual measurements produced by the operational definition, 2 general problems: k\ It's easy for operational definitions operational definitions often include\ to leave out important components extra components that aren't part of\ of a construct the construct being measured [Using operational definitions ] - Best method of determining how a variable should be measured = consulting previous research involving the same variable\ =\> you discover that standard, generally accepted measurement procedure has already been developed - Use conventional method of defining and measuring variables\ =\> results will be directly comparable to the results obtained in past research 3.2 validity and reliability of measurement =========================================== 2 criteria for evaluating the quality of a measurement procedure: validity and reliability [Consistency of a relationship ] - Validity and reliability of measurements are established by demonstrating the consistency of a relationship between 2 different measurements =\> to show the amount of consistency, the 2 scores can be presented in a graph called a scatter plot: Correlation near +1.00 correlation near -1.00 correlation near 0.00 - Reliability or validity of a measurement procedure is usually established with consistent + or -- relationship, depending on how the variables are defined and measured\ =\> correlations often used to determine validity and reliability [Criteria 1: validity of measurement] - To establish validity, you must demonstrate that the measurement procedure is actually measuring what it claims to be measuring - Question of validity especially important whenever operational definition is used to measure a hypothetical construct - Validity of measurement procedure = degree to which measurement process measures the variable that it claims to measure - Some of the most commonly used definitions of validity: +-----------------------+-----------------------+-----------------------+ | Face validity | Concurrent validity | Predictive validity | +=======================+=======================+=======================+ | \- concerns the | = validity of new | = when the | | surface appearance, | measurement is | measurement of a | | or face value, of a | established by | construct accurately | | measurement procedure | demonstrating that | predicts behaviour | | | the scores obtained | | | \- Its based on | from the new | | | subjective judgement | measurement technique | | | and difficult to | are directly related | | | quantify | to the scores | | | | obtained from | | | ! social | another, | | | desirability! | better-established | | | | procedure for | | | | measuring the same | | | | variable | | | | | | | | \- it establishes | | | | consistency between 2 | | | | different procedures | | | | for measuring the | | | | same variable, | | | | suggesting that the 2 | | | | procedures are | | | | measuring the same | | | | thing | | | | | | | | \- if 2 sets of | | | | measurement are | | | | related, it doesn't | | | | mean that they are | | | | identical | | +-----------------------+-----------------------+-----------------------+ +-----------------------+-----------------------+-----------------------+ | Construct validity | Convergent validity | Divergent validity | +=======================+=======================+=======================+ | \- past research has | \- demonstrate a | | | demonstrated how the | combination of | | | specific variable | convergent and | | | behaves | divergent validity | | | | | | | = if we can | | | | demonstrate that | | | | measurements of a | | | | variable behave in | | | | exactly the same way | | | | as the variable | | | | itself | | | +-----------------------+-----------------------+-----------------------+ | | = involves creating 2 | = involves | | | different methods for | demonstrating that | | | measuring the same | we're measuring 1 | | | construct, and then | specific construct | | | showing that the 2 | and not combining 2 | | | methods produce | different constructs | | | strongly related | in the same | | | scores\ | measurement process\ | | | =\> goal: demonstrate | =\> goal: | | | that different | differentiate between | | | measurement | 2 conceptually | | | procedures join on | distinct construct by | | | the same construct | measuring both | | | | constructs and then | | | | showing that there's | | | | little or no | | | | relationship between | | | | the 2 measurements | +-----------------------+-----------------------+-----------------------+ [Criteria 2: reliability of measurements] - A measurement procedure is reliable if it produces identical results when it's used repeatedly to measure the same individual under the same conditions\ it's the stability or consistency of the measurements produced by a specific measurement procedure - Concept of reliability: based on assumption that the variable being measured is stable or constant Changes randomly from 1 measurement to the next\ (over series of measurements: should average 0) Small error component scores will be large error component huge differences\ relatively consistent (=\> reliable) between measurements (=\> not reliable) The more common sources of error: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Observer error Environmental changes Participant changes -------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------- -------------------------------------------------------------------------------------------------- Researcher can introduce simple human error into measurement process, especially when the measurement involves a degree of human judgement Goal = measure the same individual under identical circumstances\ Participant can change between measurements\ =\> difficult, often there are small changes in environment\ =\> measurements can differ, producing what appear to be inconsistent or unreliable measurements =\> can influence the measurements ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Types and measures of reliability] Measure reliability for simultaneous measurements and measure reliability in terms of internal consistency among the many items that make up a test or questionnaire: +-----------------------+-----------------------+-----------------------+ | Successive | Simultaneous | Internal consistency | | measurements | measurements | | +=======================+=======================+=======================+ | test-retest | Inter-rater | Split-half | | reliability\ | reliability | reliability | | = established by | | | | comparing the scores | = the degree of | = obtained by | | obtained from 2 | agreement between 2 | splitting the items | | successive | observers who | on a questionnaire or | | measurements of the | simultaneously record | test in half, | | same individuals and | measurements of the | computing separate | | calculating a | behaviours | score for each half, | | correlation between | | and then calculating | | the 2 sets of scores | | the degree of | | | | consistency between | | parallel-forms | | the 2 scores for a | | reliability | | group of participants | | | | | | = if alternative | | | | versions of measuring | | | | instrument are used | | | | for the 2 | | | | measurements | | | +-----------------------+-----------------------+-----------------------+ [The relationship between reliability and validity] - Reliability is a prerequisite for validity\ =\> a measurement procedure can't be valid unless it's reliable, but a measure can be reliable without being valid - In situations with an established standard for measurement units\ =\> possible to define accuracy of measurement process\ Degree to which measurement conforms to established standard - Biased measurement = biased assessment\ =\> bias signals that the inaccuracy is predictable rather than unpredictable such as when there are 'random' errors in measurement 3.3 scales of measurement ========================= Measurement = procedure for classifying individuals into categories\ =\> the set of categories = scale of measurement -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Nominal scale Ordinal scale Interval scale Ratio scale --------------------------------------------------------------------------------------------------------------------------- ----------------------------------------------------------------------------------------------------------------- ------------------------------------------------------ --------------------------------------------- Qualitative differences in the variable measured\ Categories have different names + are organized in ordered series (consists of series of ranks)\ All categories are the same size\ =\> categories have different names but aren't related\ =\> directional relationship between the categories\ =\> consists of series of equal intervals\ =\> measurements allow us to determine whether 2 individuals are the same or different (but no quantitative comparison!!) =\> determine whether 2 individuals are different + direction of difference (but not magnitude of difference!!) =\> determine distance between 2 points on the scale Has arbitrary zero point\ Not-arbitrary zero point\ =\> 0 = matter of convenience or reference =\> 0 = none of the variable measured\ =\> measure absolute amount of variable\ =\> compare measurements in terms of ratios -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Dealing with equivocal measurements] - Some measurements aren't obviously in one category or the other - Common for researchers in behavioural sciences to measure variables using rating scales [Selecting a scale of measurement] - One obvious factor that differentiates the 4 types: their ability to compare different measurements\ nominal: tells us a difference exists\ ordinal: tells us direction of difference\ interval or ratio: determines direction and magnitude of difference\ k 3.4 modalities of measurement ============================= +-----------------------+-----------------------+-----------------------+ | Self-report measures | Physiological | Behavioural measures | | | measures | | +=======================+=======================+=======================+ | Most direct way to | Involve brain-imaging | Provide researchers | | assess a construct\ | techniques such as | with vast number of | | =\> each individual | PET, MRI, EEG, and | options\ | | is in a unique | MEG\ | =\> possible to | | position of | =\> allow researchers | select behaviours | | self-knowledge and | to monitor activity | that seem to be best | | self-awareness (no | levels in specific | for defining and | | one knows more about | areas of the brain | measuring the | | the individual's fear | during different | construct\ | | than the individual)\ | kinds of activity | =\> behaviour may be | | =\> direct question | | actual variable of | | and its answer have | \+ extremely | interest\ | | more face validity | objective | =\> constructs reveal | | than measuring some | | themselves in overt | | other response | \- require equipment | behaviours that can | | influenced by fear | that may be expensive | be observed and | | | or unavailable\ | measured (behaviour | | \- easy to distort | (+ creates unnatural | can be natural or | | self-report measures | situation for | structured) | | | participant -\> other | | | | reaction)\ | \- behaviour may be | | | =\> valid measure of | temporary or | | | construct? | situational indicator | | | | of an underlying | | | | construct | +-----------------------+-----------------------+-----------------------+ 3.5 other aspects of measurement ================================ [Multiple measures] - Use 2 or more different procedures to measure the same variable\ + provides more confidence in validity of measurements\ - statistical analysis (complex and not well understood) and interpretation of results\ - 2 measures may not behave in the same way - One method for limiting problems = combine measures into single score for each individual [Sensitivity and range effects] - Researcher begins study with expectation of how the variables will behave\ =\> concern: measurements are sensitive enough to respond to the type and magnitude of changes that are expected\ =\> sensitivity problem when obtained scores tend to cluster at one end of the scale\ c Range effect ceiling effect floor effect\ = measurement procedure = clustering of scores = clustering of scores\ is insensitive to changes at the high end of a at the low end of a\ that may occur in one measurement scale, measurement scale,\ direction allowing little or no allowing little or\ possibility of increases possibility of\ in value decreases in value - Range effects suggest basic incompatibility between procedure and individuals measured [Artefacts: experimenter bias and participant reactivity] - Artefact = non-natural feature accidently introduced into something being observed (in context of research study: external factor that may influence or distort the measurements)\ =\> can threaten validity (you're not really measuring what you intended) and reliability +-------------+-------------+-------------+-------------+-------------+ | Paralinguis | Kinaestheti | Verbal | Misjudgemen | Not | | tic | c | reinforceme | t | recording | | cues | cues | nt | of | participant | | | | of expected | participant | 's | | = | = body | or desired | 's | responses | | variations | posture or | responses | responses | accurately | | in tone of | facial | | in the | in the | | voice\ | expressions | | direction | direction | | =\> | | | of the | of the | | influence | | | expected | expected or | | participant | | | results | desired | | s | | | | results | | to give | | | | | | expected or | | | | | | desired | | | | | | results | | | | | +-------------+-------------+-------------+-------------+-------------+ - Researcher doesn't obtain valid measurements + it undermines reliability - Limit it by\ - standardizing or automating the experiment (limit personal contact between experimenter and participant)\ - using 'single-blind research' (using someone who doesn't know the expected results)\ - using 'double-blind research (neither the researcher nor the participants know exactly who is getting what until the study is completed) - In many experiments: participants don't know the hypothesis - Study pre-registration: involves specifying details of a study before it's conducted - Researcher gives directions and participant follows instructions, 2 problems:\ - participants try to figure out purpose of study =\> modify their responses to fit their perception of the researcher's goals\ - participants can become so dedicated to performing well that they do things they would never do in a normal situation - Subject roles = subject role behaviours = different ways of responding to experimental cues based on whatever they judge to be an appropriate role in the situation +-----------------+-----------------+-----------------+-----------------+ | Good subject | Negativistic | Apprehensive | Faithful | | role | subject role | subject role | subject role | +=================+=================+=================+=================+ | participants | Participants | Participants | Participants | | have identified | have identified | are overly | attempt to | | hypothesis and | hypothesis and | concerned that | follow | | are trying to | are trying to | their | instructions to | | produce | act contrary to | performance | the letter and | | responses that | the | will be used to | avoid acting on | | support the | investigator's | evaluate their | any suspicions | | investigator's | hypothesis\ | abilities or | they have about | | hypothesis\ | =\> we don't | personal | the purpose of | | =\> we don't | want this | characteristics | the study | | want this | | \ | | | | | =\> we don't | (people who | | | | want this | help science | | | | | and people who | | | | | are apathetic)\ | | | | | =\> we want | | | | | this | +-----------------+-----------------+-----------------+-----------------+ - Reactivity = problem in studies conducted in laboratory\ =\> field study: participants are observed in their natural environment and are much likely to know that they're being investigated - Laboratory = any setting that's obviously devoted to the discipline of science\ a field = a place that the participant or subject perceives as a natural environment [Selecting a measurement procedure] - First: review past research reports involving the variables or constructs to be examined\ =\> examine the options and determine which method is best [Measurement across cultures] - Some words: difficult to translate\ + languages vary greatly in the number of colour words that they have - Use the forward-backward translation procedure:\ 1 translator translates language for original measure into 'new' language, then different translator takes newly translated text + translates it back into original language for measure\ if the 2 are identical or barely differ: measure has likely not been greatly changed by the process of translation\ if they're different: warning translation may be changing the nature of the measure