Week 4 Lecture.pdf

CVEN4405 Human Factors in Civil and Transport Engineering Week 4 Lecture Neuroscience of road safety Dr Milad Haghani School of Civil and Environmental Engineering Economic aspects of road safety Cognitive aspects of road safety Mathematical modelling of road safety Clinical and epidemiologica...

CVEN4405 Human Factors in Civil and Transport Engineering Week 4 Lecture Neuroscience of road safety Dr Milad Haghani School of Civil and Environmental Engineering Economic aspects of road safety Cognitive aspects of road safety Mathematical modelling of road safety Clinical and epidemiological aspects of road safety Neuro-cognitive aspects of road safety Social science of road safety Social psychological aspects of road safety Philosophical aspects of road safety Micro-mobility aspects of road safety • • • • • Week 4: Neuro-cognitive aspects of road safety Driver visual attention and eye tracking Driver brain activity Driver state and mental workload Driver fatigue Driver mental health Driver visual attention • Eyes provide access to the inner workings of the mind and brain. • When and where drivers look is of vital importance to driver safety. • Lee (2008) reviewed 50 years of research and concluded that collisions occur because drivers “fail to look at the right thing at the right time” (p. 525). • What can eye tracking methodologies tell us about driver visual attention Eye movements consist of two primary events: • Fixations: Fixations are periods of relative stability, during which the eyes focus on something in the visual scene. • Saccades: Rapid, ballistic jumps of the eye that separate the fixations and serve to orient the focus of the eyes from one point of interest to another. No visual information is taken in during these rapid movements. Driver visual attention Panel a reflects a typical experienced driver, panel b reflects a typical novice. Driver visual attention: the effect of driving experience in modifying eye movement strategies • • • • They recorded the eye movements of driving instructors and learner drivers while they drove three virtual routes that included day, night and rain routes in a driving simulator. Driving instructors had an increased sampling rate, shorter processing time and broader scanning of the road than learner drivers. This broader scanning of the road could be possibly explained by the mirror inspection pattern which revealed that driving instructors fixated more on the side mirrors than learner drivers. Poor visibility conditions, especially rain, decrease the effectiveness of drivers’ visual search. Driver visual attention Eye-tracking glasses give researchers the possibility to capture real, objective and deep insight into human visual behaviour in real environments, more specifically by capturing their eye movement and gaze. The glasses track the exact fixations of a person in real time, which shows their focus and attention, while simultaneously enabling a person to freely move and function as usual. Since the equipment is light and designed for the unobstructed viewing of the environment, the glasses have the potential to ensure natural behaviour of the user and thus provide valid and relevant results. • • • • A sample of 19–76 years old, performed ten kilometers of urban driving in real environments, in which they were visually challenged with 56 traffic signs and 31 advertisements. To examine whether roadside object detection changes with age, the volunteers were divided into three age groups: Young (up to 25 years), mid- age (from 26 to 64 years) and older (above 65 years). Driver age was not connected with the number of detected roadside elements. Drivers who detected more traffic signs also detected more roadside advertisements. Driver visual attention: The effect of billboards on driver distraction • • • This study utilises a customised driving simulator and synchronised electroencephalography (EEG) and eye tracking system to investigate the cognitive processes relating to the processing of driver visual information. The study compares the driver’s cognitive responses to fixations on billboards with fixations on the vehicle dashboard. The experimental results demonstrate that the proposed measurement system is a valid tool in assessing driver cognition and suggests the cognitive level of engagement to the billboard is likely to be a precursor to driver distraction. Driver visual attention • • • • • • The objective of this study was to verify the effectiveness of eye-tacking metrics in indicating driver’s mental workload in semi-autonomous driving when the driver is engaged in different non-driving related tasks. In a driving simulator experiment, 36 participants experienced Levels 0, 1, and 2 automated driving while engaging in either visual-verbal, auditory-spatial, or auditory-verbal tasks. The driver’s mental workload was measured through the subjective rating of secondary-task performance and eye-tracking metrics. Different semi- autonomous levels were found to place different mental workload on the driver in both visual and auditory multi-tasking situations. Eye-tracking metrics (pupil diameter change, number of saccades, saccade duration, fixation duration, and 3D gaze entropy) were proven through correlation-matrix calculation and principal-component extraction to be effective indicators for mental-workload level prediction in both visual and auditory multi-tasking situations. These results can be used to develop an adaptive multi-modal interface that issues efficient and safe takeover requests. • • • • • Week 4: Neuro-cognitive aspects of road safety Driver visual attention and eye tracking Driver brain activity Driver state and mental workload Driver fatigue Driver mental health Neuroimaging methods in road safety research Applications of neuroimaging methods have substantially contributed to the scientific understanding of human factors during driving by providing a deeper insight into the neuro-cognitive aspects of driver brain. This has been achieved by conducting simulated (and occasionally, field) driving experiments while collecting driver brain signals of various types. Four major brain imaging methods: • • • • Functional Magnetic Resonance Imaging (fMRI), Electroencephalography (EEG) Functional Near-Infrared Spectroscopy (fNIRS) Magnetoencephalography (MEG) What is Brain Activity? https://www.youtube.com/watch?v=kMKc8nfPATI First 7 minutes Neuroimaging methods in road safety research Functional Magnetic Resonance Imaging (fMRI) • A method for depicting changes in deoxyhemoglobin concentration consequent to task-induced or spontaneous modulation of neural metabolism. • Established in 1990s, this method has been widely utilised in numerous cognitive, clinical and behavioural studies and since 2001 has been adopted to learn about driver’s brain activity. • The method was developed to demonstrate regional, time varying changes in brain metabolism and relies on Blood Oxygen Level Dependent (BOLD) signal. • It is based on the premise that cerebral blood flow and neuronal activation are coupled: when an area of the brain is in use, blood flow to that region also increases. • The method requires that subjects be placed motion-less in an MRI machine as they perform a given task. https://www.youtube.com/watch?v=rJjHjnzmvDI https://www.youtube.com/watch?v=cVeXbWCvVXY https://www.youtube.com/watch?v=BmQR57V5TVU Mins 25-35 Watch in your own time. Not examined. Neuroimaging methods in road safety research Electroencephalography (EEG) • A method to record electrical activity in the brain by measuring voltage fluctuations of the ionic current within neurons of the brain. • This electrical activity is recorded over a period of time by multiple electrodes placed on the scalp. • The method predates fMRI by a long time and has been in use since the 1930′ s. • Applications of this method in driving behaviour research were reported as early as 1978. https://www.youtube.com/watch?v=tZcKT4l_JZk https://www.youtube.com/watch?v=XMizSSOejg0 https://www.youtube.com/watch?v=pYEgal17W7k Watch in your own time. Not examined. Neuroimaging methods in road safety research Functional near-infrared spectroscopy (fNIRS) • A method that basically uses NIRS for functional neuroimaging and captures the changes in optical properties of brain tissue. • Using this method cerebral hemodynamic responses are measured by near-infrared light propagating through the head and gathering information about volume, oxygenation and flow of blood. • A sensor is attached to the subject’s forehead and connects directly to a computer. • It can also connect to a portable computing device that records the signals as the subject performs given tasks https://www.youtube.com/watch?v=y_mTFjNN5dc Neuroimaging methods in road safety research Magnetoencephalography (MEG) • Another functional neuroimaging method that records small magnetic fields produced in the brain. • Like fMRI, the method requires a scanning machine, but unlike fMRI, an MEG scanner does not emit radiation or magnetic fields Various neuroimaging methods each offer certain possibilities, advantages and limitations. Issues such as cost of operation, mobility of equipment, degree of invasiveness of experiments, confinement of subjects during tasks, preparation time for experiments, sensitivity of data to subject’s body movement, and the temporal and spatial resolution of the collected brain signal make certain brain imaging methods more suitable for certain research questions/applications in driving contexts. • EEG and fNIRS have mobile equipment whereas fMRI and MEG experiments require fixed scanners that are not portable. • fMRI and MEG also require that the participant’s head be confined in a small space and their data are highly sensitive to the head movement compared to EEG and fNIRS • MEG and fMRI are impossible for field driving experiments. • fMRI and MEG studies are expensive experiments due to the specialty facilities and equipment that they require and the high operational costs. • MEG and fMRI data have a very high spatial resolution (in the order of millimetres) thus suitable for more accurate localisation of various brain functions during driving, as opposed to the EEG method for example that has a spatial resolution of centimetres. • EEG signal, however, has a high temporal resolution and it can accurately capture changes in brain electrical activity that occur quickly. Poor temporal resolution is a clear disadvantage of fMRI, as a method that offers a delayed representation of cortical activity. • EEG is also a fairly and viably non-invasive and inexpensive method compared to the other alternative methods. • Although fNIRS equipment is technically portable and although it does not impose strict head movement constraint, in terms of the invasiveness, it is deemed as a relatively uncomfortable method given that it requires attachment of probes on the scalp. But, since it does not expose subjects to magnetic fields, it can accommodate participants with ferromagnetic implants without any safety concern. Let’s review these features fMRI Mobile equipment No need for Head confinement Not too sensitive to Head movement Suitable for field experiments Low cost Non-invasiveness & comfort Allows ferromagnetic implants High spatial resolution High temporal resolution EEG fNIRS MEG Brain activity during driving Discussion: How can brain imaging methods assist in understanding driving behaviour? Consider the case of drowsy driving. Brain activity during driving Discussion: How can brain imaging methods assist in understanding driving behaviour? Consider the case of distracted driving. Mind wandering Auditory distraction Brain activity during driving Discussion: How can brain imaging methods assist in understanding driving behaviour? Consider the case of drivers with brain damage/lesion. • • • • • Week 4: Neuro-cognitive aspects of road safety Driver visual attention and eye tracking Driver brain activity Driver state and mental workload Driver fatigue Driver mental health Driver mental workload The use of driver support systems has increased: • Advanced Cruise Control (ACC) -- Removes the mental load of maintaining a certain speed and keeps a safe distance from a lead vehicle • Lane keeping Systems (LKS) -- takes care of lateral positioning As more systems are introduced, more tasks are being taken over by automation. As a result, the role of the driver is slowly shifting from operator of the vehicle to a supervisory role (which is extremely monotonous). Driver mental workload • Even though the task demands and mental workload are reduced, supervising a (partly) automated vehicle without an active role has a negative effect on driver state. • From a safety perspective, the priority is to keep driver workload in the “optimal performance” window. • Another potential risk is introduced by the temporary increase in task demand created by these systems themselves (blind spot warning, forward distance warning, change in status of the system.) • With an increased number of systems, communication demands also increase. • Different signals can be confusing or contradictory, leading to increased mental workload. • Another increase in workload is caused by the fact that the driver has to be continuously aware whether systems are on or off and understand the systems’ limitations—and check whether these may apply. Driver mental workload The use of driver support systems and partly self-driving vehicles impact task demand, leading to possible overload or underload, two situations that impact driver state and task demands. Every level of automation should be designed in such a way that the optimal performance window is respected. Driver mental workload There are three types of measures that can provide an indication of mental workload and driver state: • Performance (primary task performance is reflected in lateral and longitudinal control) • Self-reporting (e.g., Rating Scale Mental Effort) • Psychophysiology (e.g., heart rate and heart rate variability (Peripheral Nervous System), EEG (Central Nervous System activity)) Driver state: Performance • An assessment of mental workload and driver state should start with driving performance. • In driving, primary task performance is reflected in lateral and longitudinal control. • Both lateral control, that is the lateral lane position, and its variability (standard deviation of the lateral position: SDLP) have been used for a long time to reflect driver state. SDLP is sensitive to many factors, including the use of sedative drugs such as alcohol, distraction, and fatigue. Mean lateral position can reflect strategic choices: it has been shown that drivers who noticed the sedative effects of medicinal drugs drove closer to the emergency lane, where more space is available. • Longitudinal control is reflected in the vehicle’s speed and speed variability and distance to other vehicles (time headway). The latter measure is always in interaction with other vehicles and frequently a car following test is used where a lead car changes speed and needs to be followed at a close but safe distance. In this test, driver response time (delay in sensitivity to speed changes) can be calculated. Driver state: Self-reports • Self-reports are also useful for assessing driver state. • Fatigue and sleepiness can be scored on a validated scale such as the Karolinska Sleepiness Scale. • A self-report of driver state is not enough, nor is it always accurate. Simply put, if drivers were able to evaluate their state correctly, why would accidents with drivers who fall asleep still happen? Another disadvantage of self-reports is timing: either normal behavior needs to be interrupted to obtain a rating, or ratings have to be given in retrospect after completing a (part of the) drive, which means they may be impacted by memory decay. Driver state: psychophysiological measures • The ability to register how the driver responds physically to certain situations makes psychophysiological measures very attractive as indicators of mental workload or driver state. • There is a broad range of measures, some requiring more advanced equipment than others. Heart rate and heart rate variability are relatively easy to assess, even though for the latter higher measurement accuracy is required. Average heart rate alone can reflect increased effort investment and reduced driver state. • Measures that reflect Central Nervous System activity, such as EEG (electroencephalogram), are more accurate, but they are also more intrusive than Peripheral Nervous System activity measures, such as electro dermal activity and heart rate. https://www.youtube.com/watch?v=1fKqOkqgCGs • • • • • Week 4: Neuro-cognitive aspects of road safety Driver visual attention and eye tracking Driver brain activity Driver state and mental workload Driver fatigue Driver mental health Driver fatigue • Fatigue impairs cognitive and physical performance and so greatly reduces the ability to drive safely and efficiently. • Fatigue is reportedly one of the leading causes of crashes worldwide. • The first problem is that there is no internationally recognised or accepted definition of fatigue. Fatigue is a concept far broader than sleepiness—at essence it is about performance, and the competence of a driver to respond appropriately to the demands of the drive. Fatigue is seldom recorded as a contributory factor in road crash reports, unless it is very clearly evidenced in the crash forensics, or strongly suspected by the investigating officer. Such cases are rare. Driver fatigue • The second problem with fatigue, which is that the physical manifestations of fatigue (generally identified as sleepiness) are difficult to detect without the use of invasive, high-tech, and resource-intensive measures, such as electroencephalography (EEG) or pupillometry. https://www.youtube.com/watch?v=mnQrZnCwoE4 https://www.youtube.com/watch?v=Qp5mAKQ7KtQ Driver fatigue • One legal proxy currently utilized for fatigue is number of hours driven (or flown). • In road transport however, it is difficult for traffic enforcement officials to enforce the number of hours a driver has been driving and it is often impossible to use it as a basis for prosecution. Proving the link between lack of sleep and performance impairment is very difficult. • Fatigue is often designated a contributory factor in crashes only when other factors have been eliminated. In fatal crashes, for example, fatigued drivers are commonly identified only by a lack of evidence of other causal factors, as well as no indication that evasive action had been taken by the driver. • “Maggie’s law” (New Jersey, US) was the first piece of legislation to be passed to actively facilitate the prosecution of fatigued drivers involved in crashes. Under this law, a sleep-deprived driver can be certified as “reckless” and can subsequently be convicted of vehicular homicide. Maggie’s law was based on scientific evidence that sleep deprivation has direct and measurable effects of a driver’s competence. Research has demonstrated that being awake for 18 hours produces impairment equivalent to a blood alcohol concentration (BAC) of 0.05%. At 24 hours the equivalence is around 0.1% Comparative effects of alcohol and prolonged wakefulness on five measures of simulated driving performance (mean±S.E.M.). The 0.00% BAC condition in the present study was aligned with the 24:00 h test time (16 h of wakefulness). For 0.05 and 0.08% BAC, the most comparable test times in the Arnedt and MacLean (1996) study were 02:30 h (18.5 h of wakefulness) and 05:00 h (21 h of wakefulness), respectively. Causes of fatigue • Sleep deprivation • Circadian rhythm • Chronic sleep disorders (e.g., obstructive sleep apnea, restless leg syndrome, narcolepsy and chronic fatigue syndrome) • Length of drive • Monotonous drive • Medication and drugs • Shift schedule A circadian rhythm, or circadian cycle, is a natural, internal process that regulates the sleep–wake cycle and repeats roughly every 24 hours. These 24-hour rhythms are driven by a circadian clock, and they have been widely observed in animals, plants, fungi and cyanobacteria. https://www.youtube.com/watch?v=2BoLqqNuqwA Susceptibility A few groups of drivers are more susceptible to fatigue than others. They include: • Professional drivers—not only during their driving shifts, but also driving home at the end of the shift. • Drivers who work varying shift patterns. • People who work more than 72 hours a week. • Drivers who drive at times when their circadian rhythm suggests they should be resting. • Long distance drivers, especially where insufficient rest stops have been taken. Diagnosis and legislation • Post crash (lack of evidence for other causes) • Pre-crash (time without sleep) • In the lab: ❖ percentage of eye closure ❖ ocular parameters (pupil constriction) ❖ heart rate variability ❖ neurological parameters ❖ head nodding/movement Diagnosis and legislation Fatigue related legislation does exist in a small number of countries, and examples from the United States, Europe, and Australia. • UK: Drivers of public or goods vehicles: Subject to the provisions of this section, no driver shall drive a vehicle to which this Part of this Act applies unless-There is installed in the vehicle in the prescribed place and manner equipment for recording information as to the use of the vehicle, being equipment of such type or design as may be prescribed or approved by the Minister for the purposes of this section; and That equipment is in working order. • Australia: National Heavy Vehicle Regulator (NHVR) https://www.youtube.com/watch?v=8LcUStWtzlg Engineering solutions • • • • • Week 4: Neuro-cognitive aspects of road safety Driver visual attention and eye tracking Driver brain activity Driver state and mental workload Driver fatigue Driver mental health Drivers with ADHD/Depression Attention Deficit Hyperactivity Disorder (ADHD) ADHD = cerebral dysfunction which involves problems with concentration and impulse control in about one half of adults who were diagnosed as a child. • The exist three sub-groups ▪ predominantly hyperactive (HD) ▪ predominantly with attention problems (AD) ▪ both symptoms (ADHD) • In the 1987 revision of Diagnostic and Statistical Manual of Mental Disorders DSM (DSM-IV) the notation attention deficit/hyperactivity disorder was introduced. Drivers with ADHD/Depression Attention Deficit Hyperactivity Disorder (ADHD) • Drivers with ADHD had three to four times more accidents compared to drivers without AD/HD (Barkley et al., 1993). • The diagnosis of ADHD is sometimes accompanied by the diagnoses of Oppositional Defiant Disorder (ODD) and Conduct Disorder (CD), often described as comorbidity or comorbid states; it is unclear if, or how, these states may contribute to road accidents involving drivers with ADHD. Drivers with ADHD/Depression Latest meta-analysis on ADHD and accident risk • Relative risk is 1.23 for ADHD drivers, same as for drivers with cardiovascular diseases. • The long-lasting assertion that “ADHD-drivers have an almost fourfold risk of accident compared to non-ADHD-drivers” was rebutted; this estimate was associated with comorbid Oppositional Defiant Disorder (ODD) and/or Conduct Disorder (CD), not with ADHD per se, but the assertion incorrectly maintained for two decades. Drivers with ADHD/Depression Drivers with ADHD vs drivers with depression • US Strategic Highway Research Program Naturalistic Driving Study • Groups were defined by Barkley ADHD and psychiatric diagnosis questionnaires • Included ADHD (n=275), Depression (n=251), and Healthy Control (n=1828) • Primary outcomes included self-reported traffic collisions, moving violations, collisionrelated injuries, and collision fault (3 years) Drivers with ADHD/Depression Outcomes • ADHD is associated with multiple collisions, multiple violations, and collision fault • Depression is associated with self-reported injury following a collision Relative risk for traffic violations/vehicle collisions for drivers with ADHD and Depression Drivers with ADHD/Depression Outcomes • ADHD is associated with multiple collisions, multiple violations, and collision fault. • Depression is associated with self-reported injury following a collision. Relative risk for collision-related injury and collision fault among drivers reporting at least one collision (last 3 years) Drivers with ADHD/Depression Drivers with ADHD vs drivers with depression • A probability-based sample of 3226 drivers from six U.S. sites • Subsamples with self-reported ADHD (n=274) and depression (n=251) • Vehicles outfitted with sophisticated data acquisition technologies for 1–2 years. • Crashes and near-crashes were objectively identified via software-based algorithms (blinded to clinical status). Drivers with ADHD/Depression Outcomes • ADHD symptoms portended 5% increased crash risk per increase in symptom severity score (IRR=1.05). • This risk corresponded to approximately 1 biennial crash and 1 annual near-crash per driver with ADHD. Crash and near-crash risk as a function of BAQS ADHD symptom severity scores Drivers with ADHD/Depression Crash and near-crash risk as a function of clinical group and exposure (miles driven per year). Drivers with ADHD/Depression Further reading • • • • • Week 4: Neuro-cognitive aspects of road safety Driver visual attention and eye tracking Driver brain activity Driver state and mental workload Driver fatigue Driver mental health Main references • Bowden, V. K., Loft, S., Wilson, M. D., Howard, J., & Visser, T. A. (2019). The long road home from distraction: Investigating the time-course of distraction recovery in driving. Accident Analysis & Prevention, 124, 23-32. • Crundall, D., & Underwood, G. (2011). Chapter 11 - Visual Attention While Driving: Measures of Eye Movements Used in Driving Research. In B. E. Porter (Ed.), Handbook of Traffic Psychology (pp. 137-148). San Diego: Academic Press. • de Waard, D., & van Nes, N. (2021). Driver State and Mental Workload. In R. Vickerman (Ed.), International Encyclopedia of Transportation (pp. 216-220). Oxford: Elsevier. • Dunaway, K., Will, K. E., & Sabo, C. S. (2011). Chapter 17 - Alcohol-Impaired Driving. In B. E. Porter (Ed.), Handbook of Traffic Psychology (pp. 231-248). San Diego: Academic Press. • Elvik, R. (2021). Drugs, Illicit, and Prescription. In R. Vickerman (Ed.), International Encyclopedia of Transportation (pp. 221-227). Oxford: Elsevier. • Fell, J. C. (2021). Transport Safety and Security: Alcohol. In R. Vickerman (Ed.), International Encyclopedia of Transportation (pp. 40-52). Oxford: Elsevier. • May, J. F. (2011). Chapter 21 - Driver Fatigue. In B. E. Porter (Ed.), Handbook of Traffic Psychology (pp. 287-297). San Diego: Academic Press. • Oviedo-Trespalacios, O., & Regan, M. A. (2021). Driver Distraction. In R. Vickerman (Ed.), International Encyclopedia of Transportation (pp. 113-120). Oxford: Elsevier. • Regan, M. A., & Hallett, C. (2011). Chapter 20 - Driver Distraction: Definition, Mechanisms, Effects, and Mitigation. In B. E. Porter (Ed.), Handbook of Traffic Psychology (pp. 275-286). San Diego: Academic Press. • Sinclair, P. M., & Swart, E. (2021). Sleep-Related Issues and Fatigue. In R. Vickerman (Ed.), International Encyclopedia of Transportation (pp. 611-616). Oxford: Elsevier. • Snider, J., Spence, R. J., Engler, A.-M., Moran, R., Hacker, S., Chukoskie, L., . . . Hill, L. (2021). Distraction “Hangover”: Characterization of the Delayed Return to Baseline Driving Risk After Distracting Behaviors. Human factors, 00187208211012218.