RT Assessment Reliability Issues

Testing Issues with RT Assessment

• This presentation discusses issues affecting the reliability and validity of assessments in RT (likely rehabilitation therapy).
• The core of research involves assigning numbers (or perceptions/behaviors) to occurrences.
• Accurate and precise measurement is crucial for trustworthy data.
• This involves two key processes: Conceptualization and Operationalization.

Conceptualization

• Defining a construct/concept theoretically: This involves looking at existing literature, ensuring consistency, and being guided by a researcher's theoretical framework.
• It can involve challenging existing conceptualizations.

Operationalization

• Making a concept measurable: Researchers take a theoretical definition and express it as concrete indicators/operational definitions.
• Examples are provided, e.g., "Negative social interactions" (Loy & Dattilo, 2000) which include verbal and non-verbal behaviors.

Levels of Measurement

• Nominal: Categories (e.g., gender, hair color).
• Ordinal: Ranked categories, but distances between ranks are not meaningful (e.g., class rank, satisfaction levels).
• Interval: Equal intervals between scores, but no true zero point; meaningful differences (e.g., temperature).
• Ratio: Equal intervals with a true zero point; ratios are meaningful (e.g., weight, height).

Precision of Measurement

• Higher levels of measurement provide more precision: Ratio > Interval > Ordinal > Nominal.
• Each level contains the properties of the previous level.

Examples of Measurement Levels

• Academic Year: Nominal/Ratio
• IQ Scores: Interval/Ratio
• Fatigue: Ordinal
• GPA: Interval/Ratio
• Hair Color: Nominal
• Type of Neighborhood: Nominal
• Climate: Ordinal

Measurement Threats to Validity and Reliability

• Ambiguity: Equating correlation with causation; taking something at face value.
• Apprehension/Compensation: Psychological or financial influence on the patient/client.
• Demoralization/Diffusion: Psychological states impacting responses.
• History: Events between pre- and post-measurement affecting results.
• Inadequate operationalization: Lack of clarity in definitions.
• Instrumentation: Changes in the measuring tool affecting data.
• Interaction: Outside treatments/events affecting responses.
• Maturation: Natural changes in the sample during a study affecting results.
• Mortality: Participants dropping out of a study skewing results.
• Regression: Tendency of extreme scores to move towards the average over time.
• Rivalry: Competition amongst participants impacting outcomes.
• Selection: Participants self-selecting into groups impacting data analysis.
• Setting: Contextual factors impacting the measured response.
• Treatment: Hawthorne effect; participants changing behavior due to the study itself.

Clinical Reasoning

• A cognitive process involving logical and coherent thought about possible reasons and meanings in a situation.
• It relies on expert judgment from therapists to produce the best action in specific situations.

Assessment & Codependency Issues

• Health relationships can create dependencies.
• Clients' needs can sometimes be met within the dependent relationship.
• Codependency can affect the objectivity of the therapist.

Adapting Tests

• Discusses challenges in adapting tests to different populations/settings.
• Possible issues discussed include: Misunderstanding the testing process leading to inappropriate responses, altered test environments, and other possible factors impacting the validity of the measurements.

Honesty & Deception

• Patients may consciously or unconsciously exaggerate or minimize symptoms in varying ways
• Patients may under-report or over-report symptoms to avoid appearing dishonest, minimizing negative aspects of their lives.
• Response styles may reflect limited engagement.

Social Desirability

• Clients might respond to appear better to others.
• This could manifest via denial or defensiveness.
• The Crowne-Marlowe test can detect social desirability bias through repetitive questioning and response comparison techniques.

Self-disclosure

• Difficulty sharing experiences or information with therapists.
• Potential reasons include trust issues and lack of rapport between the therapist and patient.
• This is not necessarily indicative of dishonesty.

Cultural Barriers

• Cultural beliefs, language, and social customs can create barriers to accurate assessments.
• It includes the importance of understanding the patient's needs and expectations rather than imposing a fixed set of variables or measurements.
• Language differences, misunderstandings about cultural norms, and other possible misinterpretations were discussed among possible barriers.

Summary

• External factors beyond the therapist's control can impact assessment reliability and validity.
• Recognizing potential influences in advance can minimize their effects.