Affective Computing: Introduction

Module 1: Introduction

• Affective Computing was termed by Dr. Rosalind Picard.
• Affective Computing is a computing system that relates to, arises from, or influences emotions.
• Affective Computing studies and develops systems/devices to recognize, interpret, process, and simulate human emotion.
• Affective computing is multidisciplinary, encompassing computer science, engineering, psychology, education, and neuroscience.
• Emotion influences cognition, perception, and everyday tasks.
• Affect is a stimulation state that provokes affective situations or experiences.
• Feeling is an internal perception of a situation compared with previous experiences.
• Emotion is a natural feeling originating from one's circumstance, attitude, or relation with others.
• Emotion is a visible or measurable consequence of a feeling.
• Affects generate feelings that are shown as emotions
• Emotions occur briefly, lasting a few seconds
• An emotion lasting for hours becomes a mood (Ekman, 2003).

Milestones and History

• In 1997, Dr. Rosalind Picard published her seminal book, Affective Computing.
• 2020 saw the launch of the IEEE Transactions on Affective Computing (IIEEE TAC).

Affective Computing Applications

• Affective Computing devices are used in education, security and healthcare.
• Picard (1997) describes three types of affective computing applications:
  • Systems that detect user emotions.
  • Systems that express emotions a human would perceive
  • Systems that actually ‘feel’ an emotion.
• Detection, expression, and perception are crucial in designing technologies with affective capabilities in mind.

Ethical Considerations

• A computer that can express itself emotionally might act emotionally with tragic consequences if it is engaging in certain behaviors.
• Numerous technologies have made errors costing lives, but many technologies have also saved them.
• Human safety is key when considering where to place affective reasoning in technology.

MODULE 2: Emotion Models and Measurement

Steps on how to apply emotional intelligence

• Notice when the person you're interacting with is frustrated (or showing another emotional state)
• Determine how best to respond
• Respond/make it so
• Assess how that worked
• Learn, adjust if needed for next time.
• Dimensional Emotion model (valence- arousal)

Affective Circumplex Model

• The affective circumplex depicts emotion along continuous dimensions of arousal (y-axis) and valence (x-axis) (Barret & Russell, 1998).
• Arousal ranges from calm to excited, representing the intensity of emotion.
• Valence ranges from pleasant to unpleasant.
• Discrete Emotion Model

Basic Emotions

• Six basic emotions are happiness, sadness, anger, fear, disgust, and surprise (Ekman & Friesen, 1971).
• Happiness characterizes a positive or good feeling arising from experience.
• Sadness is a passive emotion displayed by being silent/disconnected and is one of the longer-lasting emotions.
• Anger is a dangerous emotion that may cause a sudden violent reaction.
• Fear is an unpleasant emotion caused by the threat of danger, pain, or harm.
• Disgust is a negative emotion connected to human senses (taste, odor, smell, etc.).
• Surprise is initiated by a sudden, unexpected event that is short-lived.

Types of Emotional Response

• Subjective Experience
  • Experiencing emotion can be subjective, despite the widespread existence of basic emotions.
• Physiological Response
  • Emotions cause strong physiological reactions (stomach lurching from anxiety/heart palpitation with fear).
• Behavioral Response
  • Emotion is expressed and interpreted, tied to what psychologists call emotional intelligence.

Emotional Measurement

• Emotion is measured with these three response systems:
  • Subjective experience: measured with self-reporting
  • Physiological Response: measured with electrocardiogram (ECG), heart rate, blood pressure, respiration rate, blood oxygen saturation, blood glucose, skin perspiration, EEG, fMRI, PET
  • Behavioral Response: measured with vocal characteristics (amplitude, pitch, etc) and facial/body behavior.

Module 3 - Affect Detection

Modalities for Affect Detection

• Facial Expression
• Body Expression
• Speech
• Texts
• Physiological Signals

Automated Face Analysis (AFA) or Facial Expression (FER)

• Facial expression communicates emotion, intention, and physical state; it also regulates interpersonal behavior.
• AFA is leading to discoveries in identifying pain, frustration, emotion intensity, depression, and psychological distress, and reciprocity.
• Applications are emerging in instructional technology, marketing, mental health, and entertainment.

Human Observer-Based Approaches to Measurement

• Measurement can be message-based, sign-based, or dimensional.
  • Message-based Measurement
    • Observers infer emotion or affective state.
    • Assumes the face provides a direct “readout” of emotion.
    • Weakness: expressions may be posed or faked
  • Sign-based Approach
    • Uses experimental or observational methods to discover the relation between signs and emotion.
    • Used facial action coding system (FACS) - describes facial activity in terms of anatomically based action units (AUs).

Multimodal Affect Recognition

• Emotion can be detected in different modes through face, voice, gesture, posture, and physiological signals such as electroencephalography (EEG), electrocardiogram (ECG), electrodermal activity (EDA) and others.

Drawbacks of single modality or unimodal emotion recognition:

• Single modality (facial expression) is prone to social masking.
• Emotions like confusion/boredom cannot be reliably recognized from one modality (face/voice).
• Other modalities (voice, gesture, posture, contextual cues) are more prone to attempts to conceal emotions.

Multimodal Expression

• Naturalistic emotional expressions are rarely unimodal but are a complex harmony of multimodal expressions
• Multimodal affect recognition systems are consistently better than unimodal counterparts

Multimodal Datasets

• Several multimodal affect databases are publicly available.
• Three types of databases: posed, induced, and natural-emotional.
  • Posed databases: Subjects act out a specific emotion while the result is captured.
  • Induced databases: Subjects are exposed to a stimulus in a controlled setting to evoke certain emotions.
  • Natural databases : Subjects are exposed to a real-life stimulus.

Feature-Level Fusion

• Feature-level fusion is a way to create a single set from all collected features.

Decision-level Fusion

• Modalities can be independently classified using models, and results are joined using a multitude of methods.

Combination Fusion

• Combination strategies are based on voting and Prior Knowledge
  • Voting
    • A voting mechanism is the most simplest and oldest method to achieve decision-level fusion
  • Prior Knowledge
    • A weighted majority voting scheme can be used to overcome limitations of the voting approach.
    • Classifiers in this approach are not treated as equal peers, and their votes are weighted to reduce the probability of a tie
    • Weights can be calculated on the classifier's performance and error rates