Lecture Notes: Methods in Neuropsychological Research (2024/2025)

**Lecture notes Methods in neuropsychological research -- Elective course -- 2024/2025:** **Lecture 2 -- Methods in clinical neuropsychological research** **Data sharing: an open science goal!:** **Open science**: movement within scientific research to, on the one hand, make it more accessible an...

**Lecture notes Methods in neuropsychological research -- Elective course -- 2024/2025:** **Lecture 2 -- Methods in clinical neuropsychological research** **Data sharing: an open science goal!:** **Open science**: movement within scientific research to, on the one hand, make it more accessible and transparent. And if you put a lot of resources in collecting data. It is a waste to not use it to its full extent. *Advantages:* - Reduces needed resources for research. - Pooling datasets look at bigger datasets. - Allow other questions than your own to be addressed. - Make research more inclusive. - Transparency between colleagues collaboration. - Allow more collaboration. - Increase trust in science. *Many kinds of datasets:* - Questionnaire data. - Task data. - Health data. - Biomarker data. - Brain data. - Etc. *Disadvantages:* - Researchers feel ownership of their data. personal investment -- why give hard-earned data away? - Ethical issues - Fears of being scooped -- what if someone else publishes the great finding first? - Fear of errors -- what if my findings turn out to be false? - Fear of mis-use -- what if someone misrepresents my data? - Lack of time -- it's a lot of work to make a data set suitable for sharing. - Privacy concerns -- is there permission to share? But: - Research integrity and transparency should be prioritized over personal gain. - Datasets are becoming like publications -- there will be accreditation. Do people share their data when they before or after using it for their own research question mostly after the data is used to answer the own research questions. Pre-register the analysis and analyzes -- so you can show this was the intend of the study. Some datasets are created for sharing: - Biobank: any kind of biomarkers, tissue types, etc. - Naturalistic observational data, monitoring - Cross-sectional - Longitudinal. - Multi-lab collaborations. - Repositories. Some datasets are not created for sharing: - Medical / health data, registries. When no permission is given. - Scraped from data sites, social media profiles, etc. - Search engine use. - Etc. ***Only ethical if permission has been given, and no identifiable information remains!*** Example: Open science framework: open google doc, where people add a link to their data. ***Practical implications:*** - Ethical treatment has the highest priority. - Properly anonymized? - Permission given for shared use? - Increasingly requested by journals and funders. - Format sometimes prescribed, not always. E.g. open access format. - Shared datasets are a lot of work to set up - Understandable structure, good annotations. - Storage can be an issue. - Sharing platforms can provide guidance. - Using shared datasets also requires a lot of work. - E.g. cleaning/processing/selecting relevant parts. - Much easier in collaboration with the people who collected it. ***Statistical implications for shared datasets:*** - More generalizable estimate. - Depending on sampling -- usually not direct reflection of the actual population. - More power - Lower standard error. - Keep effects of significance testing in mind, i.e. in weak correlations. - Slight differences in assumptions: - Normal distributions often not as critical in large datasets. - Reduced possibility of manual checks - More (automated) cleaning necessary. - Often many missing data points. ***Observational data:*** don't intervene, manipulate - Describe research goals. - Predictive research goals. often only ethical way to research certain questions. often only way to include interactions of many influences (also pitfall). often exploratory rather than hypothesis-driven questions. little opportunity to investigate underlying mechanisms. So not large RCTs, shared labs experiments etc. (these are not observational but experimental). ***Cross-sectional datasets:*** - Between groups designs: - Making subgroups based on nominal variable. - Making subgroups based on continuous variables based on aspects of the data, or conceptual thresholds. - Making subgroups based on data-driven groupings in the data (clusters). - Within group designs - Estimating relationships between variables. - Estimating distributions of (combinations) of variables: common and uncommon occurrences. - Findings patterns, separating noise from real information. Classical ways of doing this: comparing group means, varieties of regression. More novel: machine learning methods. ***Common analyses: varies of regression:*** Main goal: estimating predictive relationships. Ordinary least squares: - Estimating the predictive value of one or more input variables on one outcome variable. - Minimizing the sum of the squared residuals. - B1 is the regression coefficient (slope) that indicates the direction and strengths of the relationship. - B0 is the intercept of the regression line. - With multiple predictors, this will be a partial regression coefficient. - The overall model accuracy is indicated with the squared multiple correlation R^2^, also indicating how much variance in the outcome variable is explained by these predictors. - Adding a 'control predictors accounts for variance in the data you might not want to test but control for (i.e. age, educational level). - Predictors that are too similar (highly correlated) to each other will not be informative in the model. Non-linear relationships: polynomial regression. Beware of overfitting -- problem where the model that is created is to complicate. It is not generalizable to other datasets. With non-parametric datasets techniques are more sensitive to latent relationships. Non-parametric methods can uncover more relationships that are not visible otherwise. ***Data-driven clustering:*** finding meaningful variance in the data without specific hypotheses. dimensionality reduction through transformations that retain the information in the data. - Exploratory factor analysis: - Describe data variability in terms of unobserved variables (factors), assumption of underlying causal structure. - Common in questionnaire development, or any data set in which multiple variables or items may measure the same construct. - Principle component analysis: - Related to factor analysis, but only correlated variance without assumption of underlying factors. - Often used to find most meaningful variance in noisy data. Model fitting: do your hypothesized factor or clusters emerge? **Longitudinal datasets:** studies that track individuals over time. - Outcome monitoring: interested in what the outcomes of particular patients are. - NESDA: Nederlandse studie naar depressieve en angst. - Others. There are various longitudinal studies that track individuals over time. - Tracking healthy individuals: - LASA: longitudinal aging study Amsterdam: healthy aging. - L-CID -- samen uniek: Leiden consortium on individual development: child development in twins. - Others. Structure of data collections in x years (study intervals). *Example longitudinal data:* aging. - Many reasons to research the effects of aging. - Personal, societal, medical. - Many individual differences. - Most notable domain for neuropsychology: cognition. - Often relation to changes in physical health, mental health, perceptual processing, lifestyle, social engagement, etc. - All these changes also impact the research options, e.g. outcome measures, design and methodology. - Many different variables potentially interact. - Important: attrition from longitudinal studies is often non-random. - Initial differences in mental/physical health status or cognitive level. - Changes in mental/physical health status or cognitive level. LASA (Hoogendijk et al., 2016) - Data collection since 1992, roughly every two years (10 full collection cycles). - Onset of cognitive impairment leads to exclusion. - Participation in experimental research studies leads to exclusion: naturalistic data! - Home visit: main interview (e.g. cognitive tests), medical interview; also online questionnaire. - Focus on physical, emotional, cognitive, and social functioning. - Large number of measurements, from neuropsychological assessments to biomarkers, health and lifestyle information. - Slide studies, e.g. addition on loneliness during covid-related lockdown. - Various others. - Goals: research, informing policy (initiated by Ministry of Health). Aging: children and adolescents: - Also many reasons to study early development. - Personal, societal, medical, and many individual differences. - Also important as predictors for later in life. - Extra complicated as development changes so many things: 'moving target'. - Extra argument for longitudinal approach. - Specific instruments and expertise are needed. - Measures adjusted to developmental stage. - Involvement of parents, additional ethical considerations. - Careful consideration of how to treat age as a covariant. - Part of national consortium on individual development. - 495 families with same-sex twins aged 3-14 years, followed over a 6-year period. - Longitudinal intervention study: not all naturalistic. - Annual assessment in various domains. - Social competence, behavioral control, cognitive measures, parenting, neurobiological and physiological measures (neuroimaging and genetics), environmental factors, family background. - Alternating home visits and lab visits. - Many fundamental questions about brain development and heritability. *Common analyses for longitudinal studies: RM-ANOVA / MANOVA:* - In cross-sectional design: comparing means. - In longitudinal designs: repeated measures. Advantages: - Easier to use, simpler techniques. Problems: - Analysis cannot handle missing values well. - Cannot easily handle slightly varying timepoints of data collection. - Cannot include multiple measure per ppt (e.g. trial-level data instead of trial means). - Only broad/general statements on development of mean scores. *Becoming more common analysis: multi-level models:* - Same concepts as regression, but accounting for clustering/covariance in the data. - In cross-sectional studies, there might also be groupings that you want to account for. - Clinical multi-center studies. - Different clinicians. - Different school - Etc Multi-level approaches: - Can uncover patterns that are otherwise obscured. - **Data visualization really helps!** - Can the clusters be interpreted? - Is there a reason why they might show differences in relationships? By combining different techniques, different patterns in the data might be found. important to relate these decisions to the content of the variables. **Big data:** - Volume: large datasets - Velocity: high speed processing (necessary for these volumes). - Variety: different kinds of data. **Machine learning:** field of research, seen as part of AI. - Key technique: classification - Training a computer model to make predictions based on available data. - Various learning methods: supervised learning, unsupervised learning, reinforcement learning, deep learning. - Key concepts in classification: training data and testing data. - Training data: used to create the model, estimate the best fit that still appears to be generalizable. - Testing data: used to test the model -- is it indeed generalizable? - Often using partitions of the same dataset. - Many application areas of this technique. - Who registry of RCTs: looking for patterns across many studies. **New directions in clinical neuropsychology:** - As computerized testing and other technological advances progress, there is greater need to integrate different data types. - Neuropsychological tests. - Biological information. - Demographics, lifestyle, etc. - Machine learning techniques can synthesize large datasets into concrete predictions. - Probably not as good as an experienced clinician at a single-patient level, but larger tendencies can be identified. - Various areas of neuropsychological research and clinical practice can benefit. **Lecture 3 -- Individual assessment:** **Phinaeus Gage (1823-1860):** - 1848, explosion during construction work in which an iron bar shot trough his scalp. - Implication: - Link between brain and personality. Main thing observed. - Long list of behavior changes reported: - Distorted, - Exaggerated. - Misinterpreted. - Untrue. - Observations: pre vs. post injury. **Need of quality assessment.** **Single cases in neuropsychology:** *Focus on the individual:* origin of neuropsychology started with focusing on the individual. How can we explain the behavior measured? **Memory:** The most striking cognitive domain in which we see changes due to brain injury. - 19^th^ century: - Importance of amnesia for theoretical insight into normal memory. - Brain is modular. - Early 20^th^ century: - Experimental work on localization in rodents. Didn't specify a memory region. - 1957 case HM. - Bilateral medial temporal lobe resection to retrieve epilepsy. - Profound forgetfulness (anterograde and retrograde). Absent of any intellectual or perceptual problems. Only memory was effected. - This told something about localization of memory. - 1980: - After development of animal model of amnesia. Role of hippocampus, entorhinal, perihinal and parahippocampal cortices were pointed out as being important for memory in rodents. - 1978- RB: lesions were found related to memory loss. Lesions of CA1 (bottleneck in processing chain hippocampus). - hippocampal damage can produce memory impairment. - additional damage in adjacent regions can exacerbate the impairment. All these observations also led to theoretical progress: - Additional cases were observed in 1990s. - Opkomt MRI en andere beeldvormingstechnieken. - Detailed neuropsychological testing testing tools improving. - Thorough neurohistoloigcal analysis in animal research. Single cell recordings and micro lesions. What did we see in these particular cases?: - HM case: the working memory is intact. - 3 items remembered is the working memory - 4 or more items to remember, this is the long-term memory that is tested. **Recognition vs recall:** group of 7 hippocampal patients. Asked eighter to make old/new decision, vs free recall (tell me what you saw).  ***Principles of memory after HM:*** - Memory is not unitary: amnesic, but able to acquire motor skill. - Modular brain: memory regions not involved intellectual or perceptual functions. - Immediate memory and maintenance are separate: amnesic, but sustained memory and rehearsal intact. - Permanent memory stored itself is elsewhere: good memory for facts and events remote from surgery. **Statistics:** ***Crawford statistics:*** how can you approach single case statistics? ***Development since 1998:*** - Discussion in publications. - Original idea: treat control sample as statistics: - Finetuning. - Additional features. - Multiple tasks: if you can find a double dissociation this can give a lot of information. - Dissociation of performance. - Here Crawford came in: the criteria should be that the difference between tests should be statistically different to be able to have a dissociation. One test should be different to controls, other task similar to controls. Afbeelding met tekst, Lettertype, schermopname, diagram Automatisch gegenereerde beschrijving - Considerations for approaching single case studies with Crawford: - Power is inevitable low. - Now you need at least samples \>10. - Power is higher when task show strong correlation. - Substantial individual differences in premorbid functioning exist. - Matching controls on multiple factors. - Further applications of Crawford approach: - Performance as regression lines. - Comparison to predicted scores - Age - Previous performance if available. - Comparison two cases and control sample. - Including covariate. **Practical approach:** ***Example NC (van der Ham et al., 2013):*** Self report complaints: - Difficulty writing in smaller boxes. - Uses post-it notes to avoid this. - Neuropsychological assessment recommended. Clinical observation: spatial problems: - Difficult to pinpoint on one particular thing. - Diverse performance on standardized visuospatial tasks. Question: what is the precise nature of NS's problem? (being unable to write in tiny boxes). NC: - Female, 25 years old. - Susac's syndrome (already known): - Rare microangiopathy: encephalopathy, hearing loss, and retical occlusions. - White matter lesions, mostly right hemisphere and corpus callosum. Assessment steps taken: 1. Regular assessment: a. Impaired on JuLO test (3/15) (judgement of lines orientation test). b. Rey complex figure test observation scored in normal range but way of drawing line lengths were off. Trying to correct it i. Corrections line length. ii. Slow processing speed. iii. Below average working memory span. 2. Task selection: c. Repetition previous tasks (5 months later). iv. Full JuLO (not short one). v. Alternative complex figure test (parallel version of Rey complex figure test). d. Spatial relation processing: vi. Categorical and coordinate performance. vii. Local and global processing. e. Tactile spatial relation task: viii. Sensory specificity ix. Calipers on hand (slide vs distance). f. Visual object and space perception test: x. Exclude visual agnosia xi. General special performance measure. g. Writing and drawing: xii. Practical problem. xiii. Writing in increasingly smaller space. xiv. Copying simple figures. - Importance of observation! - Categorical representation concerns e.g. chair - is left of the couch etc. - Coordinate representation takes into account different differences. This chair is closer to the couch than the other chair. - Left peripheral region for categorical decisions - Right peripheral regions for coordinate decisions. 3. Step 3: control participants - Participants taken from existing datasets. - Newly collected data 4. Set 4: data collection: 3. separate sessions. - Many more tasks -- exploration. - Color categorization - Progressive task selection - Be clear about the goal and content. - Ask for informed consent. - JuLO impaired: only for the oblique lines impaired. - VOSP largely norma: no visual agnosia. - Observation: focus on parts of object. triggered to do the local global test with the letters. - Spatial relations slower: opposite pattern for low/high context and coordinate processing. 5. Conclusion: - Mixed pattern of impaired performance. - Important to take into account observations. - Important to look at relative performance - High vs low context. - Local vs global processing. **Case IS:** Scientific outreach on navigation research. Participant emailed: - Unable to navigate, her whole life. - No neurological condition. Most likely: unable to create mental representations of spatial situations. Ideas: - Developmental topographical disorientation no spatial information what so ever, names rooms just in order. - Definition: getting lost daily or often (1 to 5 times a week) in the most familiar surroundings, such as their neighborhood and /or their own house. - Report this difficulty since childhood. - No memory complaints or other cognitive difficulties that may affect daily life activities. - No known brain injury, malformation, or condition affective the central nervous system, with the exception of migraine. Approach: - Brief email contact. - Ethical approval. - Task selection: - Standardized. - Experimental. - Interview questions. - Home visit. - Findings - Clarification for participant. - Possibly scientific importance. Findings: - Standardized tasks: - No visuospatial problems. - No memory problems. - No spatial location memory problems. - No evidence for malingering. - Experimental tasks: - Only problem with map creation and use, but not for learnt locations (topography of NL). - Drawing intact, except for 3d objects. - Interview: - Very strong impact on daily life. - Lifelong anxiety and dependence on others. Needed direct visual input to make a drawing of the living room, no spatial feature for the kitchen. Navigation tests taken: Findings: - Most likely developmental topographical disorientation. - Using smaller experimental control groups. - Larger normative datasets. - Missed in previous psychological help. - Knowledge dissemination important: Volkskrant newspaper article. - Many responses of people who recognize this. Future steps after SI-case: - **Clinical perspective:** support training opportunities. - **Diagnostic approach:** relation to other atypical spatial phenomena. **Assessing change in an individual:** Application: - Multiple measurements within an individual: identification of meaningful change. - Track recovery after stroke. - Disease progression. - Forensic evaluation. - (pathological) aging. Assessment: actual change + error. - Contribution of error: - Patient: fatigue. - Testing situation: lighting. - Test: administration error. Patient variables: - Demographic variables: e.g. improvement expected for age group. - Clinical condition: may vary at different timepoints. - Prior experiences: exposed to related activities affective performance. Testing situation: - Retest interval: reliability decreases with longer intervals. - Regression to the mean: drift towards population mean in retest. Test variables: - Reliability: test-retest reliability coefficient. - Practice effects: declarative and procedural memory. - Novelty: discussion -- increase of decrease in performance? - Floor and ceiling effects: consider baseline. Assessment methods: - Simple discrepancy score: T2-T1. Does not control for any of the problems discussed. - Standard deviation index: T2-T1/S1 z-score. Widely used, easy, more precise estimate, no control for variables affecting repeated assessment. - Reliability change index: simple discrepancy score / SED. Z-score T2-T1/SED. More precise estimate of change, controls for reliability of the test, does not control for practice effects. - Reliability change index + practice effects: **RCI in elderly (Stein et al., 2010):** Cognitive decline with age, looking at rate of progression in dementia. Literature review. RCI scores for instruments: - Range of neuropsychological tests and substests included. - Study quality score determined. - Sample characteristics. - Test intervals. - RCI (per age group / baseline / interval). Findings: - Only 11 studies suitable. - Long prodromal periods in dementia: - Long interval and multiple measurements necessary. - Potentially adjust RCI when intervals are short. - Cognitively impaired participants not always excluded this should be, is very important. **Lecture 4 -- Meta analysis** Power is related to significant testing Larger sample size: more opportunity to cancel out random variations. Suffer less from random variations. More robust estimate of the effect size. If we do meta-analysis, we are looking at multiple numbers for each studies, how much should I care for the effect size of study X compared with effect size of study Y,Z etc. How do I do that, by accounting for differences in sample size. Sample size is an indicator of precision of the estimate of the used in the average score. It goes from a sum or mean score of numbers to a weighted mean score. How does that work? Larger studies get more weight in the mean score. What the weights are dependent on more than just sample size. ***Meta-analysis in our case is computing a mean score of study effect sizes while accounting for differences in samples between the studies.*** **The significance of replication**: scatter plot; dots are effect sizes of study on (X-axes), and there is a replication study on Y-axes. Relying on 1 study is not a good idea. Replication always gives different outcomes. You can combine a series of first of second run studies. Instead of replicating single studies, combine different studies that individually might suffer from bias, but combined they cancel out each other's bias in meta-analysis, on average. *Meta-analysis versus review:* - Both systematically summarize literature, and discuss similarities and differences. - Review: narrative, interpretation. - Meta-analysis adds numerical synthesis (statistics). **Process model of replications***:* ***Invoegen slide*** What is happening in certain fields, there are several meta-analyses on the same topic, with different outcomes and conclusions. Leading to analysis over different meta-analysis (replications of meta-analysis). Could be useful to update to the current state of the art literature, and conclusions. You can see a trend over time. **Research literature review***:* - A systematic, explicit and reproducible method for identifying, evaluating, and synthesizing the existence body of completed and recorded work produced by researchers. - A literature review can be divided into 7 stages - Selecting research question(s): guiding the review. - Selecting bibliographic databases: + reference lists of already retrieved articles. - Choosing search terms (key words): research question search terms you use to retrieve appropriate articles. - Applying practical screening criteria: criteria for inclusion in and exclusion from the literature review population, time period, etc. - Applying methodological screening criteria: quality of research (what is a good paper/ bad paper). - Doing the review: standardized forms to extract "data" from the selected articles. - Synthesizing the results: descriptively; interpretations of the review's findings based on the reviewer's experience and the quality and content of the available literature. **Why do a literature review?:** - Intellectual: because you want to understand what is currently known about a topic. - Practical: because you need to know what is known about a specific topic in order to plan your own research. **Descriptive (or narrative) reviews:** - Reviewers use their knowledge and experience to synthesize the literature by evaluating similarities and differences in: - Purposes - Methods and - Findings of high-quality research. - Validity depends on - The reviewer's expertise and - Critical imagination and on the quality of the available literature. **What does one study tell use?** - Logical impossibility of proof. - Practical impossibility of disproof. **Why do a meta analysis:** - **Power:** combination of studies does increase power. - **Reliability increase:** due to cancelling out random variation in single studies. - **Noise reduction.** - **Practical design issues in individual studies.** **Goals of meta-analysis:** - Estimate "true" population effect. - Each study provides only an estimation, so not sure about the true effect estimate variability. - Explain variability: moderation. **Meta-analysis: procedure:** **!Invoegen slide!** **Statistical issues: overview:** - **How to combine different outcomes?** - **Effect sizes (ES):** measure of association where sample size does NOT play a role anymore. - Means and standard deviations cohen's d - Correlations association based/variance based. Not different measures, but strength of the association. Partial Eta squares - Binary data (proportions success/failure) odds ratio, chi-square. Based on categories. - Mean, SD Cohen's d / Hedges' g (small N). - Correlations Pearson r / Fisher's z - Binary data risk ratio / odds ratio - **Conversion between effect sizes:** - p z - z r - r d **Goals of meta-analysis: statistics:** - estimate combined ES: weighted average; central tendency. - Estimate confidence interval around ES - Variability within studies: s^2^ - Variability between studies: t^2^ - Explain variability between studies: - Moderators? **Estimating the population ES:** - **Theoretical models:** - **Fixed-effect model:** - Study weight: w=1/s^2^ determined by sample size - The study gets more important if the variance goes down. - The larger variance means 1/variance becomes something small. - Larger samples give higher weights. - There is only one true ES, and all studies provide an estimation of that ES. - **Random-effects model:** most used in comparison to fixed-effect model. - Study weight: w = 1/(s^2^ + t^2^) not only determined by sample size but also by the differences between individual studies. - t 2 the variability between study effect sizes - Assume there could be more than 1 true ES in the population. (There is more than one true ES, and different studies estimate different ES. The ES is of the subgroups). - Q-statistic chi-square, df = k-1 - When p \< a heterogeneity. - When p \> a [≠]{.math.inline} homogeneity. **Explaining heterogeneity:** - Moderators (study characterisitics). - Categorical ANOVA - Need at least 4 or 5 studies per group. - F-value (standard ANOVA) Q. - Numerical meta-regression. **Publication bias:** analyzing statistics from papers found in the literature. Very often papers with result are tended to be published. Papers with no sig results tend to not be published. This means there is problem because only the results that were interesting are represented. Your estimating a to high number. How high?: - **Fail-safe number**: the number of studies with average N and on average zero ES needed to reduce the combined ES to insignificant (Rosenthal). - Criterion: 5k+10 Number of studies average N and specified average ES needed to reduce the combined ES to a trivial level (Orwin). Example: if you have 10 studies, and need 100 to come to the countereffect than you probably have a strong outcome of your study. Larger fail-safe number is good. Tell you have a very robust prediction. - **Funnel plot:** plots ES(x-axes) against SE (y-axes). - Symmetric distribution around mean ES. - More sampling variation assumed in x for smaller y. - Asymmetry in funnel is a sign of missing studies aka publication bias. - Smaller studies are less precise. - Most extreme ES are left unpublished.  - **Trim-and-fill procedure.**