Scale Development and Validation

• Scale development and validation are crucial across health, social, and behavioral sciences
• There is a wealth of research on the subject, but the number of techniques needed can be onerous, jargon-filled, unfamiliar, and resource-intensive
• It is often not a part of graduate training, this primer concisely reviews the scale development process in as straightforward a manner as possible for developing new, valid, and reliable scales and help improve existing ones

Overview

• This primer provides best practices for scale development in measuring complex phenomena

• Not a systematic review, but the amalgamation of technical literature and lessons learned from experiences creating or adapting scales over several decades There are three phases with nine steps:

• Phase 1: items are generated and the validity of their content is assessed

• Phase 2: the scale is constructed, which includes pre-testing the questions, administering the survey, reducing the number of items, and understanding how many factors the scale captures

• Phase 3: scale evaluation, the number of dimensions are tested, reliability is tested, and validity is assessed

• Scales measure behaviors, attitudes, and hypothetical scenarios we expect to exist as a result of our theoretical understanding of the world, but cannot assess directly

• Scales are typically used to capture a behavior, a feeling, or an action that cannot be captured in a single variable or item

• Using multiple items to measure an underlying latent construct can additionally account for, and isolate, item-specific measurement error

• Scales lead to more accurate research findings

• Thousands of scales have been developed that can measure a range of social, psychological, and health behaviors and experiences

• New scales become necessary as science advances and novel research questions are put forth

• There are many steps to scale development, significant jargon within these techniques, can be costly and time consuming, and complex statistical analysis is often required

• Many health and behavioral science degrees do not include training on scale development

• There are a number of incomplete scales used to measure mental, physical, and behavioral attributes that are fundamental to our scientific inquiry

• A primer for best practices for scale development can be broadly applicable across many disciplines, especially for health, social, and behavioral sciences

• The primer is an amalgamation of technical literature and lessons learned from experiences spent creating or adapting a number of scales related to multiple disciplines

Three Phases of Scale Development

• They can be further broken down into nine steps Item development (coming up with the initial set of questions for an eventual scale) is composed of:

• Identification of the domain(s) and item generation

• Consideration of content validity

• Scale development (turning individual items into a harmonious and measuring construct), consists of: pre-testing questions, sampling and survey administration, item reduction, and extraction of latent factors

• Scale evaluation requires tests of dimensionality, tests of reliability, and tests of validity

Identification of Domain and Item Generation

• Specify the purpose of the domain, confirm that there are no existing instruments, describe the domain, specify the dimensions of the domain, and define each dimension
• Identify appropriate questions that fit the identified domain; utilize deductive methods (literature review and assessment of existing scales) and inductive methods (exploratory research methodologies including focus group discussions and interviews Evaluation by Experts to evaluate each of the items constituting the domain for, content relevance, representativeness, and technical quality
• Quantify assessments of 5-7 expert judges using formalized scaling and statistical procedures including content validity ratio, content validity index, or Cohen's coefficient alpha and conduct Delphi method with expert judges
• To evaluate each item constituting the domain for representativeness of actual experience from target population
• Conduct cognitive interviews with end users of scale items to evaluate face validity

Pre-Testing Questions

• Assesses questions to reflect whether the domain of interest and produce accurate measures
• Administer draft questions to 5-15 interviewees in 2-3 rounds while allowing respondents to verbalize the mental process entailed in providing answers

Survey Administration and Sample Size

• Gather enough data from the right people; collect data with minimum measurement errors
• Administer potential scale items on a sample that reflects range of target population using paper or device; recommended sample size is 10 respondents per survey item and/or 200-300 observations; sample size is needed to ensure the availability of sufficient data for scale development
• Ensure the availability of data for scale development and validation; use cross-sectional data for exploratory factor analysis
• For test of dimensionality (Step 7) use data from a second timepoint, at least 3 months later in a longitudinal dataset, or an independent sample

Item Reduction

• Ensure the scale is parsimonious for determining item and probability of examinee correctness
• Proportion can be calculated for CTT and item difficulty parameter estimated for IRT using statistical packages
• To determine the degree to which an item or set of test questions are measuring unitary attribute for CTT and the probability of correct response changes as ability increases for IRT
• Estimate biserial correlations or item discrimination parameter using statistical packages

Inter-Item and Item-Total Correlations

Correlations of scale items and each item along with sum score

• Estimate using statistical packages
• To determine the distribution of incorrect options and how they contribute to the quality of items
• Estimate distractor analysis using statistical packages; deleting or imputing cases
• Ensure the availability of complete cases for scale development
• Delete items with many cases that are permanently missing, or use multiple imputation or full information maximum likelihood for imputation of data

Extraction of Factors

• Explore the number of latent Constructs that fit the observed data
• To determine the optimal number of factors or domains that fit a set of items

Tests of Dimensionality and Hypothesized Latent Constructs

• Address the latent structure of scale items and their underlying relationships, i.e., validate whether the previous hypothetical structure fits the items
• Estimate independent cluster model-confirmatory factor analysis
• Estimate measurement invariance to determine whether hypothesized factor and dimension across groups or multiple samples
• Create scale scores for substantive analysis including reliability and validity of scale; calculate scale scores using both weighted and unweighted approach

Tests of Reliability Statistics

• To assess degree to which the set of items in the scale co-vary, relative to sum score
• Estimate using Cronbach's alpha; used to assess scale reliability using tests like Raykov's rho, ordinal alpha, and Revelle's beta
• Assess the degree to which the participant's performance is repeatable; i.e., how consistent across time
• Estimate the strength of the relationship between scale items over two or three time points; a variety of measures possible

Criterion Validity and Intended Latent Dimension

• Determine score prediction of future outcomes. criterion and predictive validity; estimate association between scale scores and "gold standard" of scale measurement
• For predictive validity, use bivariate and multivariable regression; stronger and significant or causal effects suggest greater; stronger significant association in Pearson product-moment correlation suggests support for concurrent validity

Construct Validity

• To examine if the same concept is measured in different ways
• Estimate the relationship between scale scores and similar constructs using multi-trait multi-method matrix, latent variable modeling, or Pearson product-moment coefficient
• To examine if the concept measured is different from other certain concepts; to examine if the concept measured behaves as expected in relation to "known groups"
• Select known binary with correlations of standard deviations of differences