FEX602 Facial Expression, Gestures & Postures PDF

FEX602. FACIAL EXPRESSION, GESTURES AND POSTURES 1 Published by Evidentia University of Behavioral & Forensic Sciences 111 E Monument Ave Kissimmee, Florida, USA © 2022 Evidentia University of Behavioral & Forensic Sciences 2 1. FAC...

FEX602. FACIAL EXPRESSION, GESTURES AND POSTURES 1 Published by Evidentia University of Behavioral & Forensic Sciences 111 E Monument Ave Kissimmee, Florida, USA © 2022 Evidentia University of Behavioral & Forensic Sciences 2 1. FACS in infants. Regarding the different methods used to code facial expression, of all the systems created by various authors, the FACS system created by Ekamn (1978) and the MAX system by Izard (1979a) have been by far the most profound and accurate. Between the two, the FACS system is more comprehensive than the MAX system (Oster, Hegley, & Nagel, 1992). The MAX system offers less description of facial action units (Malatesta, Culver, Tesman, & Shepard, 1989) and fails to differentiate some anatomically distinct expressions while considering some that are not (Oster et al., 1992). The FACS system defines AUs (Action Units) as fundamental actions of muscles or mus- cle groups. AD (Action Descriptor) are unitary movements that may involve the action of several muscle groups. The intensity of the movement is described by adding letters from A to E (min- imum intensity - maximum intensity) where A is minimum intensity (trace), B mild intensity, C marked or pronounced, D severe or extreme and E maximum. The FACS (Facial Action Coding System) is a system for coding human facial expressions. It is a common standard for systematically classifying the physical expression of emotions. This system can encode virtually any anatomically possible facial expression through its deconstruc- tion into specific action units (AUs) and their temporal segments. The AUs are independent of any interpretation, which makes the system highly useful for scientific purposes. In FACS, each AU is defined as one or several muscles that are contracted or relaxed (Fernández-Abascal & Chóliz, 2001). Later they developed (Ekman, Irwin, Rosenberg, & Hager, 1997; W. Friesen & Ekman, 1983) the EMFACS (Emotion Facial Action Coding System) and the FACSAID (Facial Action Coding System Affect Interpretation Dictionary). 3  In general terms, it was after the publication of the FACS system that the proposals of Ek- man and Friesen (1978) became truly reliable. As mentioned before, it is necessary to emphasize that the creation by the authors of a codified and specific system for measuring facial expressions was the starting point for an infinite number of new lines of research, including those presented in this study. The coding and classification systems of emotional facial expression emerged to define emotions as we saw in the previous chapter. It allows the identification of emotions based on the muscular activity involved in facial gestures and is based on the changes in appearance produced as a consequence of the visible ac- tion of each muscle. It describes distinctive actions such as skin movements, temporary changes in shape and location of features, skin folds and onset, termination, intensity and asymmetry of facial expres- sion, while ignoring non-visible changes (changes in muscle tone, superficial skin coloration, sweating, tears, rashes, etc.), which are also important in the recognition of emotions. Another very interesting evidence is the comparative analysis of the facial expression of adults and newborns. In this sense, the facial musculature existing in adult humans is the same as that existing in newborns and is fully functional at birth (Ekman & Oster, 1979). It is even interesting to look at modern 3D ultrasound scans that offer facial expressions of the unborn baby that speak for themselves: 4  For authors such as Darwing, Ekman or Izard, the study of the expression of emotions in infants could justify the innate character of this, since basic emotions are produced without being learned or the product of early experiences (Fernández & Chóliz, 2008). Fernández & Chóliz state that “it cannot be concluded that the facial expression of emo- tions is innate”, however they speak of a biological predisposition to express, by means of a cer- tain gestural configuration, certain physiological or behavioral reactions, due to the fact that we all belong to the same species and share an anatomical and physiological configuration that makes us common to all of us regardless of the ethnic group to which we belong. 2. Body orientation and body movements. Emblems and illustrators, regulators and adapters. Research on nonverbal behavior has been rather neglected in the study of posture and ges- ture, compared to areas such as facial expression or proxemia. This paucity of systematic studies contrasts with the large number of interpretations that have been made of the psychological signif- icance of posture and gesture in books and publications dealing with nonverbal communication, nonverbal language or body language. 5  Body language can reveal voluntary and involuntary, conscious and unconscious motiva- tions, attitudes, intentions and reactions.. In addition, according to Deutscheach person has a char- acteristic basic posture at rest to which he or she returns after deviation from it and may also give us much information about social relations and the structure of an interaction (Scheflen, 1964, 1973). Body movement is the central axis of kinesis (or Kinesis), the pioneer author in this field being Ray Birdwhistell (1952). Authors such as Eco and Volli (1970) defined kinesis as “the uni- verse of body postures, facial expressions, gestural behaviors, of all those phenomena that oscil- late between behavior and communication”. Thus, kinesics would include aspects such as body orientation, postures, gestures, facial expressions, eye and eyebrow movements or the direction of the gaze. The separate study of each of these aspects is probably not justified since we believe, like Patterson, that the nonverbal system works in a coordinated and integrated way, with the overall effect being greater than the sum of its parts (Patterson, 2011). However, although our approach is intended to be functional, we have broken down nonverbal behavior into different components (such as postures, gestures or facial expression), which should not be confused with trying to find meaning in nonverbal behavior based on the different elements in isolation. Let us remember that meaning should never be sought in the isolated components, but in all the combined information that comes to me through the different channels. Thus, for example, the meaning of a glance will depend on a series of different elements such as the facial expression that accompanies it, the distance from the interlocutor, the posture, the body movements that are being carried out at that moment, etc. Body posture We call posture the disposition of the body, or its parts, in relation to a given reference sys- tem (Corrace, 1980). This reference system can be another element of the same body, the rest of the body or other bodies of other individuals. Posture and gesture are intimately related since they can involve the same parts of the body. We will treat posture as a static position, since a posture endowed with movement would be treated as a gesture. 6  Erving Goffman (1961) studied the postural rules that governed the different encounters among the staff of a psychiatric hospital. He found that subjects of higher status had less rigidity of rules, presenting a wider range of postures than subjects of lower status. If we differentiate between dominant-superior and submissive-inferior postures, an upright posture, with the head thrown back (haughtiness) and the hands on the hips (desire to occupy more space), could be in- terpreted as a certain desire for domination. Normally the posture adopted is involuntary on most occasions and can intervene to a greater or lesser extent in the communication process. Some authors have shown the link between body posture and attitudes, situations and social contexts, such as Charlotte Wolf (1966), Scheflen and Scheflen (1972), Nierenberg and Calero (1976) or Ray Birdwhistell (1979). But we also know that posture and body movements are linked to the emotions we feel and that they somehow reflect them. In this sense, there is evidence that posture varies with the emotional state of the individual, especially through the tension-relaxation dimension (Mehrabi- an, 1968, 1977). For Mehrabian (1977), moreover, there is a clear relationship between posture and the attitude held towards the receiver. Thus, if the receiver is of a higher status, the individual will be more tense than if the receiver is of a lower status. For this purpose, he observed positional signs of relaxation; thus, the asymmetrical position of joints, the oblique or reclining inclination and the relaxation of hands and neck would denote positional relaxation. He also observed that physical proximity, more intense eye contact and forward leaning are signals that communicate a positive attitude towards the recipient. For their part, Ekman and Friesen (1969) pointed out that posture can reveal to others the attitude, confidence or self-image of the subjects. They also showed that posture is less controllable than the face or tone of voice, and that posture can reveal an anxiety that the facial mask does not allow to be externalized. Other authors (Trower, Bryant, and Arglyle, 1978) have argued that body posture serves to communicate different traits of the subject, such as attitudes and emotions. We see, theefore, that through body postures we could obtain valuable information about a subject’s attitude towards a possible interaction or as nonverbal indicators of the subject’s status or power. 7  Signs indicative of an open attitude towards an interaction could be reflected through pos- tures with relaxed arms, feet directed towards the person with whom we are going to interact, frontal orientation and direct eye contact; indicators of a closed or rejecting attitude could be observed in postures with closed arms, tilted body or averted gaze; indicators of power and sta- tus could be seen in expansive postures, trying to occupy more space (e.g. with arms in a jug), arrogant postures (e.g. with chin raised), proud postures (with chest puffed out and back curved backwards), etc. We could group the different postures maintained by an individual into different categories. Thus, for Albert Mehrabian (1968) there are four different postural categories: 1- Approach postures: postures that transmit attention or interest, shown by the inclination of the body forward. 2- Withdrawal postures: these would be negative postures of rejection or repulsion and would be shown by retreating or turning away. 3- Expansion postures: these would be proud, arrogant, haughty or contemptuous postures and would be shown by the expansion of the chest, an erect or leaning backward trunk, erect head and elevated shoulders; 4- Contracted postures: these would be depressed, depressed or crestfallen postures and would be communicated by a forward leaning trunk, a sunken head, slumped shoulders and sunken chest. But let us remember once again that the correct translation of most of the indicators and postural categories depends mainly on the context in which they are produced, mediated by other variables such as sex, age, race or culture of the person performing them. Moreover, these postural categories will not mean the same thing if they are performed in isolation or in front of other people. In the latter case, we should stop to assess the body orientation between subjects. 8 Body orientation By body orientation we are referring to the degree of deviation of the legs and shoulders that an individual maintains with respect to another individual, or other individuals, during an in- teraction. If the interaction is between two people, the main orientations that can be assumed are frontal orientation (“face to face”) and lateral orientation (“side to side”). According to Ricci and Cortesi (1980) body orientation could provide us with information about the relationships between two subjects, whether they are collaborative, friendly or hierarchical. If the relationship is hierar- chical, the superior subject will be positioned in front of the inferior; if the relationship is one of collaboration or close friendship, the two subjects will adopt a side-by-side position. But talking face to face, with the bodies facing each other, can also be a sign of intimacy or of not wanting to suffer interruptions in the conversation. On the other hand, when two people hold a conversation in places such as a party or a meeting, and are positioned at an angle greater than ninety degrees, they would be indicating to other people their willingness to join in the conversation. In childhood, it is likely that we come to associate the greater height of parents and adults with people in authority. Perhaps because of this, the cultural convention has been established that when one person stands taller than another, it is because he or she holds a dominant position with respect to that person. On the other hand, there are cultural differences when it comes to adopting one orientation or another. Thus, Swedes tend to avoid positioning themselves at ninety-degree angles (Ingham, 1971), or Arabs prefer to adopt face-to-face positions (Watson and Graves, 1966). Scheflen (1964) observed that changes in the orientation of body parts such as shoulders, hips or joints are slower than those of the eyes or head, highlighting the importance of these changes during any interaction. This author distinguished three groups of body orientations: - Guidelines for inclusion or non-inclusion in the group - Front, side, or angle orientations - Orientations of congruence or incongruence with the group 9  Albert Mehrabian (1971, p. 1) speaks of a principle according to which “people are attract- ed to people and things they like, evaluate highly, and prefer; and they avoid or move away from things they dislike, evaluate negatively, or do not prefer”. This author speaks of the relationship between the orientation of the body in an interaction and the attitudes or status differences be- tween the interactants. Thus, seated women assume a more positive attitude the more direct the orientation of the body is towards an imaginary individual, but in the case of very dear addressees, the orientation loses importance. In the case of men, there is no relationship between orientation and attitude towards unwanted persons, and orientation is less direct than in the case of women, only towards very wanted individuals. But the results of their experiments are probably not very valid, since they were carried out in front of imaginary interlocutors. Gesture or body movement. Gesture, as a form of movement, is distinguished from posture in that the latter refers to static positions of the body. Gesture includes not only movements of the hands and arms, but also of other parts of the body such as the head, trunk, legs or feet. The study of gestures dates back to ancient Rome, when authors such as Cicero and Quin- tilian, in their treatises on rhetoric, included a series of observations on the use of hands in oratory. The “live” observation techniques of that time have been replaced by the observation of videotape recordings and, later, by digital or DVD recordings. Such recordings allow the repeated visualiza- tion, and if necessary in slow motion, of the most complex gestural sequences, coming to perform true “microanalysis” of nonverbal behavior, examining and dissecting even the smallest detail of gestures, expressions and body movements. Thus, some authors (Bull, 2012) have even drawn a parallel between the video recorder and the microscope used in biological sciences. According to Ricci and Cortesi (1980), the various studies carried out on gestures have tried to establish a relationship between gestures and emotions, to attribute meaning to them or to analyze their functions in relation to verbal communication. Today we know that gesture is inti- mately linked to speech, but also that it plays an important role in the communication of emotions and interpersonal attitudes. Let us look at some important aspects of this link between gesture and speech. 10 Gesture and speech We could say that an individual’s body movements are closely synchronized with his or her speech (autosynchrony). But we also know that, sometimes, the body movements of two con- versational partners become synchronized (interactional synchrony), with the listener providing constant feedback to the speaker, in terms of attention level and interest. Although on this last point there is considerable controversy (Bull, 2012). For Bull, gestures relate to speech in terms of their temporal coordination (coinciding with syntactic boundaries and separating important words and phrases) and in terms of meaning (con- veying meaning through their visual appearance, as with emblems). In relation to meaning and unlike speech, gestures can simultaneously convey multidimensional aspects of meaning. Thus, for example, a powerful iconic image could convey information at the same time about the move- ment of an object, incorporating simultaneous information about speed and trajectory, whereas through speech this information would have to be conveyed sequentially. As we will see below, the types of gestures most closely linked to speech will be emblems, illustrators and regulators, but before going into typologies, let us look at the link between gesture and emotion. Gestures, emotions and interpersonal attitudes There is an enormous link between the expression of an emotion and the important role played by the face. Thanks to the work of Darwin, Tomkins, Ekman and others, it is now com- monly accepted that there are at least seven universal emotions with innate facial expressions. We also know that these emotions are manifested primarily through the face. In this sense, Ricci and Cortesi (1980) maintain that the face is a specialized communication zone that we use to commu- nicate emotions and attitudes. Ekman and Friesen (1969) affirm that the face is the primary seat of emotion expression, calling the nonverbal signals that express an emotion “affect exhibitors”. Vi- cente Caballo (1993), maintains that the face is the most important and complex area of the body in nonverbal communication, and the part that is most observed during an interaction because it is the main system of signals that we possess to show emotions. 11  But we also know that the facial expressions of emotions can be modified and controlled, showing others different from those actually felt. Thus, the expression of a genuine emotion could be attenuated, amplified, substituted or hidden. In this regard, it is interesting to note that some gestures are more difficult to control or hide than others. According to Morris (1977) we attach less importance to the deliberate control of body parts that are farther away from the face. In fact, when an individual tries to hide an emotion, he will exert more control over his facial expression, but he will be much less aware of his movements as they move away from the head. Thus, the movements and postures of more distant areas, such as the feet, will provide more valid signals when interpreting the true emotional state of an individual. In this sense, if a person is nervous and does not want to show signs of this uneasiness, he will try to simulate tranquility by controlling the verbal content of what he says and manipulating his facial expression, but the continuous move- ment of his feet could be giving him away. We will not go into the details of the gesture through facial expression as this point was developed in the previous chapter. Body postures and emotion Humans, like other animals, express power through open postures, expansive postures; powerlessness, however, is expressed through closed postures, contracted postures. But can adopt- ing postures that denote power really cause us to feel these sensations? The study of the connection between the mind and the body could be traced back to the 19th century. One of the most influential psychologists of that century in the history of psychol- ogy was William James. His work, Principles of Psychology (James, 1890), was considered one of the most representative manuals of the 19th century in psychology. William James’ theories on emotion, and on the ideomotor effect, could be considered as a clear antecedent of modern theories on the influence of the body on the mind. The ideomotor effect would be a psychological phenomenon whereby a subject involuntarily and unconsciously performs movements due to the influence of suggestion or expectation. Regarding emotions, James (1884) maintained that emo- tional experience is the result of the perception of bodily changes; both visceral and postural, as well as facial expression. 12  Following his approach, when we perceive a danger and run away, our mind realizes that we have run away to avoid the danger; it would be then when we would experience the corre- sponding emotion, which in this case would be fear. Therefore, for James, we are afraid because we run away and not the other way around; that is, the perception of bodily changes by the subject would give rise to the emotion. In other words, visceral and motor bodily reactions would be the basis of emotional states. Since then, many studies have highlighted the bidirectional relationship between nonver- bal behavior, thought, and emotions (Laird & Lacase, 2014). In a review of the research carried out on the effects of body expansion, Carney and collaborators (Carney, Cuddy, & Yap, 2015) cite at least thirty experiments, published from 1982 to the present day, that have highlighted the relationship between expansive posture and variables such as feelings of power, the intake of less food, the leader’s choice of seats, pain, a positive attitude towards oneself, risk tolerance, risk acceptance, various endocrine responses, memory biases, locus control, depression or feelings of pride, among others. Perhaps the most significant study, which gave rise to several replications, was the one carried out by Carney and colleagues (2010), which showed that poses that convey high power, unlike poses that convey the opposite, provoke neuroendocrine changes, which would affect certain psychological processes and behavior in both men and women. Thus, high power poses would provoke elevations in testosterone levels (dominance hormone) and decreases in cortisol levels (stress hormone), as well as a greater sense of power and risk tolerance; in contrast, individuals adopting low power poses would exhibit the opposite pattern. In summary, according to these authors, adopting poses of dominance and power for a few minutes provokes physiolog- ical, psychological and behavioral changes. However, in a conceptual replication of that study, conducted by Eva Ranehill et al. (2014), they only found an effect of expansive posture on subjective feelings of power, but not on risk tol- erance, testosterone, or cortisol. The apparent discrepancy in these results is explained, according to Carney et al. (2015), due to three key differences of their studies from the Ranehill et al. study: in their studies they did conceal the experimental purpose, their experiments involved a social task during the postural manipulation, and the postural manipulations used were comfortable, easy to adopt, and of short duration (lasting one minute versus three minutes in the Ranehill et al. exper- iments), avoiding causing discomfort, awkwardness, and habituation to the effects of the posture. 13  A criticism of all these experiments on the influence of power postures is the failure to de- termine, in most of these articles, whether the blinded method was used, since the impact of the experimenter and the pervasiveness of expectancy effects are key variables to be controlled. The meaning of gestures One of the pioneering studies on the meaning of gestures was carried out by Ephron (1941). David Ephron, in order to demonstrate the erroneous racist theses that existed at the time about Jews, carried out observations on the non-verbal behavior of Jews and Italians living in New York. He found that although behaviors were different in first-generation emigrants, they were uniform in their descendants, exhibiting fewer gestures specific to their group of origin. He also showed that individuals exposed to the influence of several different groups in their gestures ended up adopting and combining the gestural behaviors of those groups. Ephron came to establish a typology of gestures, a typology that served as a basis for Ek- man and Friesen (1969, 1972) to establish the well-known five categories of kinesthetic behavior, according to their origin, use and codification. It must be said that although these categories refer to the movements of all parts of the body, they especially define the gestures carried out with the hands. In this well-known typology they distinguish between emblems, illustrators, regulators, displays of affection and adapters, pointing out that these categories do not have an exclusive character; thus a gesture would not necessarily be included in only one of the categories, but could belong to more than one. The emblems Emblems are non-verbal behaviors that have a specific meaning and could be translat- ed directly into words. They are gestures emitted consciously and intentionally. The function of emblems is purely communicative and they emerge as an alternative to voice communication, to be used when speaking conditions are difficult or impossible (for example between workers in a factory with high noise levels), there is a great distance between sender and receiver (for example in the case of hunters on a hunt), or by agreement (for example in a game of mime). Also, they can be used to replace or repeat the content of a verbal communication, to give greater emphasis to some aspects of the verbal message we are trying to convey (for example if we make the emblem of the telephone, taking our hand with the little finger and thumb extended towards the mouth and ear, while we pronounce the phrase “I’ll call you back”), or in an ironic sense contradicting what was said verbally. 14  We know that the precise meaning of an emblem is known by most of the components of a group, social class, subculture or culture. Curiously, many of the existing emblems have a meaning of offense or insult (for example the typical “sleeve-cutting”) and perhaps this is because making an insult, using an emblem at a certain distance from the other interlocutor, can guarantee escape if things get ugly. According to Ekman and Friesen (1972) emblems can involve actions of any part of the body, although typically the hands, head orientation, facial musculature movements or posture are involved. People are usually as conscious of the utterance of emblems as they are of the utterance of their words, but sometimes what Ekman called “emblematic slips” can occur (Ekman and Friesen, 1972). These are situations in which an individual apparently does not deliberately choose to use an emblem but does so unconsciously. Thus, the “comb gesture”, consisting of leaving the middle finger rigid while keeping the hand closed in the form of a fist, sometimes appears unconsciously symbolizing expressions such as “fuck you”, “go to hell” or “suck it” (remember that this gesture symbolizes a penis with testicles). Another typical emblematic slip consists of making an affir- mative movement with the head while pronouncing “no” verbally (or vice versa) or shrugging the shoulders as a symbol of “impotence”. There are emblems that are specific to a particular culture, but there are also emblems that, to a greater or lesser extent, are universal or almost universal: Some of these emblems would be, for example, the well-known “eyebrow flash” (raising of the eyebrows for a very brief duration, similar to one-sixth of a second) used as a greeting; the mocking gesture, consisting of resting the thumb on the nose while the other fingers point upwards and spread out like a fan (very wide- spread in Western Europe); waving the hand, as a greeting or farewell; putting the hand to the mouth, as a sign of hunger (not in all countries); or tilting the head to one side, closing the eyes and placing the hands underneath as a pillow, as a sign of sleep. 15  The problem with the use of emblems is that the same sign can have very different mean- ings, depending on the culture in which we find ourselves. Thus, it is known as the typical “ring gesture”, where the thumb and index finger come together forming a circle, can mean that some- thing is “ok” in countries like the United States or Great Britain, “zero” or “nothing” in countries like France, “money” in countries like Japan and be an obscene sexual insult in countries like Italy or Brazil (as it would symbolize the “anus”). Perhaps for this reason, several authors have tried to draw up lists of emblems corresponding to different cultures, such as Kaulfers (1931), Green (1968), Saitz & Cervenka (1972), Johnson, Ekman & Friesen (1975) or Poyatos (1977). Illustrators Illustrators are gestures that are directly linked to speech and illustrate the content of the message or its intonation. Illustrators are gestures emitted consciously and in some cases inten- tionally. They facilitate communication by sometimes separating successive parts of speech or functioning as a punctuation system. At other times they amplify the content of the message by describing spatial relationships, or by drawing shapes of objects; but they can also contradict what is expressed verbally or be used as a substitute for a word. They differ from emblems in that most illustrators do not have a precise verbal meaning, do not appear unless accompanying the speech and are shown exclusively by the sender. For Ekman and Friesen (1972) changes in the frequency of occurrence of illustrators, in middle-class white American individuals, depend on the state of mind and existing problems in verbal communication. Thus, when a person is demoralized, discouraged, tired, unmotivated, worried about the impressions he or she will make on the other person, or has the non-dominant role in a formal interaction, the rate of illustrators will be lower than is usual for that person. On the other hand, when there is interest, enthusiasm for the topic or communication process, when one has the dominant role in a formal interaction, or in an informal interaction when there is little concern for the impressions one will make on the other person, the person will use more illustra- tors. Also the number of illustrators will increase when there are difficulties in finding the right words or when the listener’s feedback suggests that he or she is not understanding what you are trying to communicate. Illustrators have generated a great deal of research and authors such as Ephron (1941) have established a classification differentiating up to six types of illustrators: baton, ideograph, deictic, spatial, kinetograph and pictograph. Ekman and Friesen (1972), based on this classification, in- creased the types of illustrators to the following eight: 16 - Batons: movements that mark rhythmic characteristics of the message and are usually used to emphasize a word or phrase. - Ideographs: movements that graphically show the speaker’s path or line of argument. - Deictic movements: illustrators used to physically point out an object, a place or an event mentioned in the message. - Spatial movements: illustrators describing spatial relationships. - Rhythmic movements: movements that describe the rhythm or pace of an event. - Kinetographs: movements that describe a bodily action or some non-human physical ac- tion. - Pictographers: illustrators who trace the image or form of what they refer to. - Emblematic movements: emblems used to illustrate a verbal statement, either by repeating or substituting a word or phrase. Recall that the same gesture can belong to more than one category and, depending on its particular use and context, it could be considered either as an emblem or as an illustrator. More- over, illustrators can be considered as regulators in certain cases (in cases where they serve to manage the flow of a conversation). Many of the illustrators are probably acquired at a very early age, during language acqui- sition. Ekman and Friesen (1972) highlighted the important role that illustrators play in our early stages of life, in directing the attention of adults and as an aid in explaining matters for which young children have no words. They also emphasized that illustrators not only help the speaker to explain himself and the listener to understand what he is saying, but also help the speaker to get out of those situations of certain clumsiness of thought or speech, accelerating the fluency of his ideas. Subsequent studies have shown how gestural accompaniments to speech can facilitate lexical access (Rauscher, Krauss, & Chen, 1996). In their study, speakers were videotaped while describ- ing cartoon actions, whether or not they were allowed to gesture. When they could gesture they did so more often during sentences with spatial content than during sentences with other types of content, and when they could not gesture their speech with spatial content became less fluent. The limitation of gestures led to an increase in the relative frequency of pauses, due to the increased difficulty of accessing the lexicon by other means such as gestures. 17  Other authors have wondered why people cannot keep their hands still while speaking. Goldin-Meadow and coworkers (2001) propose that the gesture lightens the cognitive load while the person is thinking about what to say. To this end, they designed a study in which they asked children and adults to recall a list of words or letters while explaining how they solved a math problem. It turned out that both groups remembered significantly more items from the list when they gestured during their explanations of the math problem than when they did not, because ges- turing allowed them to save cognitive resources in the explanation task and use more resources in the memory task. Many other studies have shown how spontaneous gestures accompanying speech play an active part in learning (Singer & Goldin-Meadow, 2005), help children learn by creating new ideas about mathematics (Goldin-Meadow, Cook, & Mitchell, 2009), or that spontaneously pro- duced bystander gestures during research interviews convey substantive information and that of interviewers serve as a source of information (Broaders & Goldin-Meadow, 2010). Regulators Regulators are gestures that aim to regulate and control the flow of a conversation. They mainly regulate the synchronization of interventions throughout the conversation, and can give in- dications to the speaker as to whether or not the listener is interested in what is being transmitted, if he/she wants to intervene, or if he/she wants to interrupt his/her speech. Some studies have shown that the listener’s attempts to assume the speaking turn in a con- versation can be eliminated by the speaker’s continuous gesticulation, and that stopping gesticu- lating is one of the main keys to ceding the speaking turn to the other. Other cues in that ceding of the speaking turn could be the completion of a grammatical sentence, an increase or decrease in pitch at the end of a sentence, an accent on the final syllable or expressions by way of crutches, such as “but”, “uh” and “do you know” (Bull, 2012). Other authors add the gaze as a turn-taking signal (La France and Mayo, 1976), emphasizing that Euro-Americans tend to look directly at their interlocutors when they are receivers, as a sign of interest in the conversation, while Af- rican-Americans prefer to avert their gaze and consider the behavior of Euro-Americans as an insinuation of confrontation or confrontation. We see, therefore, that the cultural component, once again, is present when interpreting gestures. 18 Signs of affection These are mainly movements of the body and face with which we convey emotions and feelings. Although these signals of affection are mainly expressed through the face, posture or certain gestures can also reveal emotions. For Ricci and Cortesi (1980) both anxiety and emotional tension provoke recognizable changes in the movements of the subjects. Thus, for example, fist shaking would be a recognizable gesture expressing anger. Affection signals can be intentional or unintentional. A smile toward a person, or looking at someone to express affection, would be examples of intentional affective signals. Blushing, paling, or pupillary dilation would be examples of unintentional affective signals. Signals of affection are often subject to cultural norms. Thus, for example, in China the duration of these signals is much shorter than in European cultures. Adapters These are nonverbal behaviors that arise mainly to try to adapt to a situation, to satisfy bodily needs, or to handle and manage emotions as a reaction to a physical or psychological state. These self-regulatory gestures would be residual behaviors from man’s ontogenetic past, that is, vestiges that would remain of certain behavioral patterns that had an adaptive function in early life. They are usually performed unconsciously and without the intention of communicating in- formation to another person. In general, what is sought with these behaviors is to obtain security or dominance and they can be triggered by the motives or affects that are being verbalized, or by the discomfort or anxiety provoked by the situation or the conversation. If the appearance is to satisfy a bodily need, such as picking one’s nose or scratching one’s head, in private situations, without the presence of other people, according to Ekman and Friesen (1972) the adaptations will be carried out in their entirety and to their completion. 19  On the other hand, when the individual feels observed, is in the presence of another person and, particularly, when intervening in a conversation, the occurrence of adaptations is reduced or appears in a fragmented form (the hand may go towards the nose but will not go as far as inserting the finger in it; or the hair may be touched without scratching thoroughly). Ekman, in some of his publications, speaks of manipulations (Ekman, 2009, p. 114) re- ferring to “all those movements in which one part of the body massages, rubs, scratches, grabs, pokes, pinches, squeezes, accommodates or otherwise manipulates another part of the body”. He differentiates between brief manipulations, which would be performed for some purpose (such as tidying one’s hair or scratching some part of the body) from manipulations that last a long time and would be performed without any purpose (such as rubbing one finger against another or rhyth- mically tapping one’s foot against the floor indefinitely). For Ekman, manipulations can also be performed by contacting or rubbing one part of the body against another (tongue against cheek, teeth lightly biting the lip, or one leg against another). Ekman also argues that manipulations in- crease when the subject feels upset or anxious (Ekman, 2009, p. 116). Adapters or manipulators can be of three types: intrapersonal or self-adapters, interperson- al adapters or heteroadapters, and object-directed or object-adapters. Self-adaptations would be the gestures or actions that we perform on ourselves, that is, movements of manipulation of our own body. They usually appear in situations of tension or anxi- ety and are performed in an attempt to seek security in order to face this uncomfortable or stressful situation. Some examples of self-adaptations would be scratching slightly some part of the body or face, running the fingers through the hair, biting the nails, pinching, crossing arms, squeezing one hand with another, etc. Especially interesting are all those adapters that try to facilitate or block a sensory input (such as the mouth or ear). Self-adapters have received different denominations according to the authors. Rosenfeld (1966) speaks of self-manipulators, Mahl (1968) of autistic gestures and Freedman and Hoffman (1967) of body-centered gestures. 20  Authors such as Ekman and Friesen (1972, 1977) maintain that self-adaptors are indica- tors of discomfort that normally arise, and increase their appearance, when the person feels un- comfortable, awkward, tense, anxious, etc., or when the person is at ease or alone, in which case self-adaptors related to grooming appear. They also maintain (Ekman and Friesen, 1972) that spe- cific self-adaptors are related to specific feelings and attitudes. Poking or squeezing a part of the body is an aggression against oneself or others, temporarily displaced on oneself; covering one’s sight with one’s hand reveals the desire not to be seen or to hide out of shame. In a conversational situation, the individual is usually unaware of the performance of any self-adaptor and rarely receives direct attention, or any kind of commentary, from the listener (with the exception of parents to children). When an individual displays any self-adaptive, he or she breaks eye contact with the person with whom he or she is interacting until the completion of the self-adaptive. Interpersonal adapters or heteroadapters would involve prototypical interaction strategies, whether in movements of approach, courtship, intimidation, aggression, protection or flight. Some examples would be hair tying, as a seductive gesture, or tie knotting. Object adapters also usually appear to try to cope with an uncomfortable or stressful situ- ation, but they differ from self-adapters in that they are performed by manipulating objects, such as a pen, a cigarette, a box of matches, a ball of paper, etc. If the object is used to carry out an instrumental task we would speak of object adapters. Thus, scribbling with a pencil, tapping with a pen, playing with a small ball of paper or playing with a cigarette would be considered object adapters. On the other hand, taking notes during a conversation or smoking would not. We are usually less aware of making these types of adapters than we are of auto-adapters and their significance is more one of general uneasiness. Other gesture categories Other authors have proposed classifications different from the one proposed by Ekman and Friesen. Thus Rosenfeld (1966), who divides gestures into self-manipulation and gesticulation. 21  Freedman and Hoffman (1967), who classify them into movements centered on objects and related to discourse, and movements centered on the body and unrelated to discourse. Mahl (1968), who differentiates between communicative gestures and autistic gestures. And also Argyle (1975), who speaks of illustrative gestures and signs linked to language, conventional signs and sign language, movements that convey emotions and interpersonal attitudes, movements that ex- press aspects of personality, and movements used in rituals and ceremonies. Barrier adapters Hands crossed in front of the crotch (or protecting vital organs) make men feel safer when threatened. Since childhood we hide behind a barrier when we feel threatened and these behaviors of concealment and protection continue to appear in uncomfortable, stressful or risky situations. Let’s think about what soccer players unconsciously do when they form a barrier with their bodies because they are going to shoot a foul: they protect their vital organs such as genitals, heart, neck or face. The protective position consisting of placing the hands, one on top of the other, in front of the genitals, is usually a more typical gesture in men than in women, but not exclusive to them. Also, on many occasions, if a threatening situation arises, we cross our arms over our chest. Allan and Barbara Pease (2006) justify this gesture as an unconscious attempt to block what we perceive as a threat, or as unwanted circumstances, and argue that monkeys and chimpanzees also do it to protect themselves from a frontal attack. It is clear that in many occasions when a person is nervous, negative or defensive, usually adopts this gesture; but the reading we make will always depend on the context, as it may be due to very different reasons, such as feeling cold, want to hide a spot, want to highlight your biceps, enhance your chest, or may adopt it for comfort. We all, on many occasions, find it comfortable to cross our arms. But it is also true that the adapters arise, among other reasons, to try to satisfy that need to feel more comfortable (as a reaction to a physical or psychological state) and with our arms crossed we will try to look for more comfort when we have a negative, nervous or defensive attitude. 22  The greater or lesser importance of this crossing of arms will be determined by the context in which it occurs and an overall assessment of the gestures shown. This global assessment was made clear by McNeill in his work Hand and Mind (1992) when he stated that “the meaning of the parts of a gesture are determined by the whole”. In this case we should take into account what his facial expression denotes, what degree of tension is observed in his limbs, if in addition to crossing his arms he shows closed fists or, on the contrary, his fingers extended, etc. Regarding the crossing of the legs, we know that the legs have played a vital role in our survival, making it easier for us to flee, stop or attack. Well, it seems that our legs unconsciously continue to “protect” us from the most everyday “dangers”. Allan and Barbara Pease (2006) main- tain that crossed legs reveal a closed, submissive or defensive attitude; on the other hand, open or uncrossed legs indicate an open or dominant attitude. But this closure can also occur at the ankles, thereby also indicating this discomfort. Subtle adapters denoting insecurity or nervousness There are a multitude of camouflaged forms of arm folding that are carried out by people who prefer to project a calm and controlled attitude. However, in reality they are indicators of their state of nervousness or insecurity. We refer to those subtle gestures that involve crossing the arms in front of the body, such as the one made to adjust the cufflinks, a bracelet, watch, touching the button on the cuff of the jacket, or any object that is on the other arm. In the case of women, these subtle barriers are usually formed by holding a handbag, a folder or a bouquet of flowers. A mixture of insecurity and nervousness would include those self-adapters in which the hands comb the hair, touch the lips, or in which the subjects bite their fingernails, insert a finger in their mouth, etc. 23 New challenges for the analysis of nonverbal behavior and the study of human emotions. We know that human behavior can be both verbal and nonverbal (Scherer and Ellgring, 2007) and that, within nonverbal behavior, we can find paraverbal communication (sounds), facial expressions, gestures and looks, as well as body movement itself, analyzed by biomechanics and kinesiology (Hinson, 1977). Biomechanics is the science that examines the internal and external forces acting on the human body and the effects produced by them. It relies on biomedical and technological sciences. On the other hand, kinesiology is the science that studies movement. It is a set of sciences that seeks to analyze the structure and function of the musculoskeletal system of the human body (Birdwhistell, 1970). Thus, the movement of the body can be studied with special reference to both the postures of the body and the movement of the body itself, kinematics, as stated by Bull (1987). Both ac- tions are important, being measured automatically through dynamics, i.e. kinetics and statics. We can define kinematics as the branch of physics that studies the laws of motion, position and trajec- tory. And dynamics as the part of mechanics that studies the relationship of motion, including the causes that produce it. Thus, kinetics would be the motion itself and statics, the equilibrium. The measurement of body movement and its representation have been the subject of study for many centuries (Meijer, 1989). Already, standing, standing upright, as well as walking, has often been considered as an automatic or reflexively controlled motor task, i.e., a motor response generated involuntarily in reaction to a stimulus without requiring a conscious decision. The spe- cial interest in its analysis is due to the fact that both movement and body posture tell us about the action and intention of the person, his or her affectation or mental state and attitude towards the circumstance in which he or she is living (Wallbott, 1998). We see, therefore, that body movement can be conscious or unconscious (Neumann, 2002), becoming automatic. Finally, the external manifestations of body movement can be carried out individually, such as frustration or aggression, or through interaction with others, in situations of domination, turns or interruptions. 24  Similarly, we can measure body movement through hypothesized responses to nonverbal behavior, hypothesized responses to phenomena with verbal correlates, computational models of human bodily behavior, and communication control and games (Wallbott, 1998). As Deal, Montillaro and Scherer (2012a) state, the measurement of the body and its move- ment can be both qualitative (coding) and quantitative (capturing movement). Qualitative mea- surement is usually carried out manually, through observers collecting and coding specific behav- iors and with possible existing bias, and by analyzing the frequency of different behaviors over time. When coding nonverbal behaviors, we must take into account all the interpretations that can occur, as well as irrelevant movements that can and should be ignored. Fundamental is the aspect that the coding scheme must be known in advance, requiring analysis. It is important to highlight the slowness with which body coding is carried out, due to the large number of coders used, whose subjectivity and possible bias should be analyzed. We also have to keep in mind that body movement encodings are qualitative, not quantitative, so there will be differences in speed, direction and shape that should be coded additionally (Gross, Crane, & Fredrickson, 2010). The objective of body movement measurement is that of an automatic, rapid measurement, carried out with minimum effort, aiming at maximum objectivity and, therefore, as quantitative as possible. This leads us to minimize the interpretation of behavior. History of body movement The interest in current human and animal movement patterns dates back to prehistoric times, where they were drawn in caves and statues were erected to represent human and animal locomo- tion systems. Later, Aristotle (330 B.C.) and Plato were considered the fathers of biomechanics, since they wrote about body segments and animal movements and displacement. Indeed, Aristotle stud- ied different body parts and the displacement of animals. 25  He claimed that movement is produced by interaction with the ground. He wrote works such as “On the motion of animals”, “On the gait of animals” or “On the parts of animals” (Hin- son, 1977; Hutchinson, 1961). On the other hand, Leonardo da Vinci (1500) studied flight and the laws of the aerial and aquatic environment. Thus, in his Man of Vetrubio we can observe the work of the first biome- chanical scientist, where his observations of human movement surprisingly fulfilled Newton’s third law (Klette and Tee, 2008). In the 19th century, it was Eadweard Muybridge (1831-1904) who made photographs of human and animal movement. With the recording and analysis of any event (from a horse race, to the gait of a child with cerebral palsy or the performance of a high-performance athlete), biome- chanics (motion analysis) has progressed rapidly in the 20th century. Etienne Jules Marey (1830-1904) carried out chronocyclic photography of human move- ment. Several authors used the photographic rifle (1882), first the manual one and then the electric one (1899). Braune and Fischer carried out the first 3D kinematic analyses, studying human gait in different sports, between 1895 and 1904. In 1887, Moritz Benedikt used the word biomechanics for the first time in a scientific sem- inar in Germany. Later, Nicholas Berstein (1896-1966) would focus on the study of the efficiency and coordination of different movements, postulating several theories on the motor control of movements. Muybridge (1887) in the USA shot 24 cameras sequentially to record the patterns of a man walking and running. Representation of body movement Biomechanics describes movement, the adaptation to movement of the human body, by describing the characteristics of the environment and its interaction with the human being (Hin- son, 1977; Neumann, 2002). It is the study of the control of our movements and deals with the processes that occur from the moment we decide to undertake them until we activate the muscles necessary to perform them (Harrigan, 2005). 26  The body can be represented as segments and joints (Kipp, Neff, and Albrecht, 2007). Segments are the parts of the body and have a given length. Joints connect these segments. Upper limb control involves the hierarchical regulation of several body segments in which the movement of each joint is a function of the movements of other joints. Thus, movement takes place in the more than 200 joints we humans have, most of them in the back, hands and feet. All the joints together form a kinematic tree, the segments being connected by the joints at each end of the tree (Bänziger, Mortillaro, and Scherer, 2012). The root joint is at the top of the tree, these joints being the highest in the tree, which affect the continuation of the chain. The joints can move in different directions and each direction contains one degree of free- dom (DOF) and each joint can have up to 3 DOF. The joints rotate around axes and for each axis there is one possible motion feature. This range is limited by restricted rotation. The DOF, with constraints, determines the range of motion. Thus, as Birdwhistell (1970) asserts, we can specify a body position from the rotations of all the joints. When both elbows flex and extend at the same time, they are said to be in IN phase. If one flexes, while the other extends, we say that they are in antiphase (AN). In the same way we can define the relationship between elbow and wrist of the same arm: isodirectional (if both flex or extend) or non-isodirectional (if they go in opposite directions). All in all, there will be up to 8 combinations of coordination modes, some more difficult than others to execute due to natural preferences. As Bánzinger and Scherer (2010) state, when performing body movement notation, we are often interested in the location of end effectors (hands, feet or head), as the same location on a hand may have different sets of rotations. Comparing positions on the basis of joint rotations is difficult. Joint locations are points in 3D space and each point can be written as x, y or z, with distances along axes from the origin. The origin is the point 0, 0, 0, 0, therefore it is necessary to define the axes. 27  In sum, a body posture can be described by the engagement of all joints and a body movement will be produced by the change of position in a room by the movement of all joints. In this way, the initial global location will depend on the origin arranged in the room as a reference, using the position in space together with the localized body posture for this. A local joint location, i.e., relative to a part of one’s own body, can also be taken as a reference (Kipp et al., 2007; Laban, 1956, 1975). The local origin is usually located in the pelvis, which is the most frequent root joint. Thus, the location of a joint will be the location of the pelvis plus the relative location of the joints. The local loca- tion will be nothing other than the global location minus the local origin. We can see, therefore, that joint locations do not depend on the position in space, calling this position normalization. This normalization will help us to quickly compare the positions and deduce whether or not they are the same. In this way we can represent the body positions as joint articulations in 3D (Poppe, 2010), we can normalize the joint locations of global position and global orientation. It would only remain for us to calculate the differences between postures, which are necessary for the quantitative analysis of body movement and position (Sayette, 2001). As we have seen, the difference in posture lies in the distance between all pairs of joints, since the movement corresponds to the body postures over time. Thus, by averaging the distances over time, we can calculate the average movement. This can be done for the whole body or per joint/limb. Body movement: sensors and devices Body movement analyzed through sensors and devices is widely used for filmmaking and video game creation. Thus, Avatar or The Lord of the Rings used this type of techniques and means to carry out their production. There are different techniques, with and without special devices. Thus, those with special devices can be mechanical (measuring angles and distances directly), markers (visible markers on the body) and inertial (through magnetic sensors on the body). Those that do not use special devices collect the vision through a depth camera. 28  We found different models. One of the best known is the Animazoo Gypsy 5, with sensor suits that measure angles and extension. The measurement is direct and allows the control of sev- eral actors at the same time. Its main drawback is the difficulty of preparation in its use and the limited number of joints, which also limits freedom of movement. The Vicon MX uses retrore- flective markers attached to a suit or through straps. This system gives greater freedom in marker configuration. Many cameras are required for recording, with the most important limitation being that the markers can be easily interchanged or fall off. The Xsens MVN consists of sensors with gyroscopes in straps that allow the use of a large stage space, as well as outdoor recording, even with multiple actors. Depth cameras, such as the Microsoft Kinect, Asus Xtion or PrimeSense, are also used for body motion notation, avoiding on-body sensors, but having relatively low accuracy, difficulties with direct sunlight and limitations in stage space (Bente, Krämer, Petersen, and de Ruiter, 2001). BAP: Body Action and Posture System In order to perform a correct notation and analysis of body movement, we must not lose sight of the fact that the body shows us not only the emotions, but also the intensity with which these emotions are being experienced. Laban (1956, 1975), decades ago, already carried out the automatic recognition of body expressions through the study of speed, strength, smoothness and tension produced in the movement. Thus, both the posture of the body as a whole and of certain parts of the body of great importance, such as the hands, are still little researched today. And it is at this point when it is essential to analyze the body movement through its coding systems. We will have to analyze, for this purpose, each of the movements performed, separating when the movement appears, how long it lasts and how it is segmented into submovements, all for better analysis and in order to develop a coding system capable of providing systematic and reli- able descriptions of body movements, with a particular focus on the bodily expression of emotion. The BAP (Dael et al., 2012a) is a body posture coding system based on movement patterns. The system is designed by 10 actors representing a total of 17 emotions. 29  It consists of a manual in which the emotions represented are defined, as well as the differ- ent rules for performing the coding. These body movements representing the different emotions (GEMP) (Bänziger and Scherer, 2010) are recorded using the ANVIL software (Kipp, 2007), which uses a double camera, front and side, with a medium plane (up to the actor’s knee) and allows their analysis following scientific criteria (Kipp. 2007). In performing the analysis and notation of body movement we must take into account the existence of observer bias, which, in order to be as minimal as possible, must always have the manual and the definitions and understand them clearly, not coding in the case of doubts and cod- ing in an orderly manner, one joint at a time, in the case of a multi-joint movement. Thus, we must keep in mind that we will have to focus on one joint for each analysis, dif- ferentiate the actual joint from the artifact (or segment), distinguish between posture and action, and be clear about the direction of the movement, as well as the time points. We define, therefore, the posture unit as the one made up of the union of the posture of the head, trunk and arms, encoding the movements made by the joints, i.e. the neck, trunk (through the spine and chest), shoulders, elbows, wrists and fingers (for each of the arms), as well as the knees and hips for the lower extremities. Posture coded with the BAP will be as described by head orientation, trunk orientation, general posture, arm posture, and gaze. The coded behaviors will range from head movement or trunk movement to arm movement (Dael, Mortillaro, & Scherer, 2012b). To carry out the encoding, the encoders will be faced with the video images, without voice, with the front and side images of the cameras simultaneously and being reproduced at a normal speed and with a reduced speed. Thus, we will find examples of interest, irritation, despair, anxi- ety, fear, panic, anger, euphoria, joy, pride, pleasure, sadness, relief or amusement. Body movement notation is booming and research in this field is advancing by leaps and bounds (Pollick, 2001). However, much remains to be done, requiring more detailed and accurate measurements of body movement and facial expressions. 30  We must not only measure, but also understand what the body behavior is, reaching an in- creasingly quantitative analysis and even being able to anticipate the movement itself by studying the variability of behavior between subjects. These lines of research are being experimented with virtual characters and robots, with great success (Tracy and Robins, 2007). Along with the analysis of movement in robots, movies and video games, the future of body movement notation is to achieve a correct analysis of it and the emotions conveyed by it. We must not forget that posture and body movement tell us as much about the action as about the intention of the person, his affectation or mental state and his attitude towards the circumstance he is living. In this aspect, the investigation of the interpretations of body movement in different cultures can help us to establish integral safety programs, which are fundamental for wellbeing in today’s globalized world. 31