Test Construction: PSMM-6 Overview

Okay, here are detailed study notes based on the provided text.

Course Overview: Test Construction (PSMM-6)

• First lecture introduces test use, and assignment details.
• Course name is 'Test construction', course code 'PSMM 6'
• Learning goals: Knowledge of test and questionnaire construction principles; Understanding effective construction, evaluation, interpretation for aims and groups; Knowledge of score use.
• Topics include test construction, psychometric properties, item response models, validity, norm-referencing and test use.
• Lectures are held once a week on Wednesdays from 9am to 11am.

Course Resources

• Course information and manual are available on Brightspace.

Achieving Success in the Course

• Sincere advice includes preparing for each lecture by reading specific topics.
• Reviewing materials after each lecture is also advised.

Psychological and Educational Tests: Components

• Test construction involves development and application, including determining the test's appearance, administration details, scoring, and interpretation.
• The administration provides usefulness to individuals or policy and what information they offer, and also how to combine scorings.
• Test theory utilizes statistical analysis related to item and test scores.
• Requires statistical theories about behavior of item and test scores such as classical test theory, and item response theory.
• Important issues include quantitative measures for items and tests aimed towards target groups.
• Both test construction and test theory are essential for sensible test utilization.

Practical Applications of Tests

• Human resource management uses tests for personnel selection and development.
• Education utilizes tests for student development and performance evaluation.
• Clinical psychology, neuropsychology, and developmental psychology apply tests to psychodiagnostics.
• Tests facilitate judgments about both communities and individuals.

Research Applications of Tests

• Hypothesis testing and theory building are key research applications.
• Variables such as indicators and size/location of brain damage + behavioral difficulties like anxiety or lack of insight determine type and severity of behavioral difficulties.
• This can be done on individual or community level.

Defining Psychological and Educational Tests

• A psychological or educational test serves as a tool to measure maximum potential under certain conditions.
• Performance is assumed to reflect latent attributes.

Standarization

• Fixed test conditions are emphasized
• Examples include fixed test material instructions and specific testing conditions.
• The main objective is comparability.
• Perfect standardization is hard.
• Standardization depends on test and target population.

Test Types

• Typical performance: typifies person; no correct answers (personality, attitude, mental health)
• Maximum performance: achievement (intelligence, ability level).
• Distinction in maximum tests include power tests without time limits; limited time tests; speed tests that are focused on timing of item answers

Latent Attributes

• Directly unmeasurable attributes like verbal ability and depression severity.
• The test score should reflect the true score/latent attribute of interest.
• A relationship exists between the attribute and the test score as persons differing on an attribute get a different test score

Important Terminology

• Item: smallest test unit on which a person's response is scored. The score can be the person's response.
• Subtest: independent test portion indicative of an attribute and composed of items.

Test Construction Steps

• Define the construct of interest first.
• Secondly, develop the test
• After the first 2 steps, complete a pilot study, and analyze / collect data
• Finish off by validating the test and working on norming.

Test Score Judgments

• Experimental findings note judgements on tests are important.
• Sarbin's 1943 Admission setting studied predictions of data/admission with extra interview info.
• Adding information reduced predictions in study and increased errors.
• Statistical methods often outperform human judgement since people minimize errors.

Improving Judgments

• Expert judgment is inconsistent with decision-making. and often replaceable by simple rules.
• Difficult improving decisions is by is including additional interaction terms.
• The process to realize this means using a consistent rule based approach.
• This reduces 'random fluctuations' and enhances reliability/consistency by using independent judgments.
• Then you need to take the median/mean, famous experiment to use is Galton's "Wisdom of the Crowd",

Reducing Error Further

• Understanding what people claim/do is a prediction matching the outcome.
• So the coherence between facts and judgments is needed to reduce error, a process producing good judgements over similar cases.
• Do not make stories based on feeling.

About Assignment

• The core of assignment is experiencing decision with tests/interview scores to see "overconfidence" in action.

Defining the Construct

• Includes abstract concepts and literature reviews.
• Considers the number of latent attributes (dimensionality).
• 2 Types: unidimensional versus multidimensional.

Test Development Components

• Measurement mode identifies how to properly develop a test based on the method.
• Defining the objective is needed
• Target specific test responses that are well written.
• Administration modes are needed

SDQ specific example

• Strengths and Weaknesses Questionaire screens behaviour within 2-17 yr olds.
• SDQ is available in many languages
• SDQ consists of 25 items on psychological attributes: Emotional symptoms, conduct/hyperactivity, peer problems, and prosocial scales
• It's resulting scores show prosocial qualities which indicate problems.

Types of version available for SDQ

• Various self-report and other-report types.

Developing the SDQ

• The objective is research versus practise.
• The population is either group level or individual level.
• Decide for diagnosis depending on if it is test or descritopn.
• Administration: a paper pencil form is useful or use different computerized programs.

Important parts of test development

• A conceptual framework helps to write times.
• Typical performance includes deductive, deductive, and intuitive classes.
• An internal based strategy is Factor Analysis.

Factors for developing tests

• Response mode should many (be looked around in box).
• Used scales should be dichotomous or ordinal.

Aspects of item writting

• Represents one idean and should be specific.
• Consdier reading level of users and avoid negative sentences.

About pilot study

• check what instrucitons and item are clear.
• There are experts pilots to understand.
• Raters need to yield info to remove or make new.

Interrater Agreement

• 2 Different raters assess subjects(items) while 1 rater assesses 2 things twice which shows consitency.

Meausres of Agreements per Scale Type

• Measures exist for all different scales.

P0

• It misleads.
• High by chance.

Coefficient

• Expresses agreement of raters.

Marginal Frequencies

• To correct for chance.

To compute expected frequency do ...

Kappa value "corrects" to show chance to agree

• Ranges to -1 to 1 with 1 showing perfect and -1 showing not perfect

Kappa is Useful

• Useful based on how many possible responses two raters offer.

Next is finding the summation operator

Then comes Alternative intra-class correlation

We can analyze Assignments in ICC