Eye Contact and Video-Mediated Communication: A Review PDF
Document Details
Uploaded by Deleted User
2013
Leanne S. Bohannon,Andrew M. Herbert,Jeff B. Pelz,Esa M. Rantanen
Tags
Related
Summary
This article reviews research on how eye contact affects communication in video-mediated interactions. The authors explore how video conferencing, compared to other methods, impacts verbal and nonverbal elements, and perceptions among participants. The research delves into the biological and cultural factors influencing gaze interactions, and implications for the quality of communication.
Full Transcript
Displays 34 (2013) 177–185 Contents lists available at SciVerse ScienceDirect Displays journal homepage: www.elsev...
Displays 34 (2013) 177–185 Contents lists available at SciVerse ScienceDirect Displays journal homepage: www.elsevier.com/locate/displa Eye contact and video-mediated communication: A review Leanne S. Bohannon a,⇑, Andrew M. Herbert b, Jeff B. Pelz c, Esa M. Rantanen d,1 a Rochester Institute of Technology, 18 Lomb Memorial Drive, Rochester, NY 14623, United States b Rochester Institute of Technology, Department of Psychology, 2323 Eastman Hall , 18 Lomb Memorial Drive, Rochester, NY 14623, United States c Rochester Institute of Technology, Carlson Center for Imaging Science, 3112 Carlson Hall, 54 Lomb Memorial Drive, Rochester, NY 14623, United States d Rochester Institute of Technology, Department of Psychology, 2353 Eastman Hall, 18 Lomb Memorial Drive, Rochester, NY 14623, United States a r t i c l e i n f o a b s t r a c t Article history: A relatively new form of human communication, video-conferencing has become more popular as video Available online 23 November 2012 technology improves and with increasing demands for real-time communication across greater distances. The full effects of video-conferencing on human communication are still being explored. Video-confer- Keywords: encing is presumed to be a somewhat richer form of communication than email and telephone, but Video-conferencing not quite as informative as face-to-face communication. This review explores research into the influence Eye contact of eye contact on communication and how video-conferencing mediates both verbal and non-verbal Communication interactions. Facilitation of eye contact is a challenge that must be addressed so that video-conferencing Eye-tracking Display can approach the rich interactions of face-to-face communication. Laptop Ó 2012 Elsevier B.V. All rights reserved. 1. Introduction Along a continuum of communication media and their ability to manage rich information, or information that most reduces uncer- In the history of human communication, video-conferencing is a tainty and equivocation, face-to-face communication is the richest relatively new means of correspondence dating from the introduc- medium that provides immediate feedback and conveys many cues tion of the AT&T Picturephone at the New York World’s Fair in in the form of natural language. Video-conferencing falls below 1964. However, the technology has advanced in leaps and face-to-face communication but above the telephone in terms of bounds in the last 10 years, making videoconferencing available information richness. The importance of the view of the interlocu- to anyone who can afford a modern laptop computer or a smart- tor’s face is exemplified by the finding that including a still photo- phone. In comparison to email, instant messaging and other elec- graph to electronic mail can enhance the intended message and tronically mediated communication, video-conferencing did not even prompt compliance with it. Human communication is become commonplace until very recently. It is now standard for complex and based upon a combination of verbal and non-verbal laptop computers and tablets to come equipped with a small cam- cues to exchange information. Eye contact is arguably one of the era mounted above the monitor, Apple’s iPhones feature an appli- most important non-verbal cues in communication, and we will cation called FaceTime, Skype (video-conferencing software) has examine its role in video-conferencing systems. been purchased by Microsoft, and Cisco (a company that special- The unique properties of video-conferencing can impact our izes in remote communication) has introduced u mi, a video-con- visual and verbal communication along with our perceptions of ferencing system designed for home use. The recent revolution one another. Users of such systems also show substantial could also be driven in part by an increasingly global economy preference for video-conferencing that facilitates eye contact. , family migration patterns, faster network connections, and Understanding these effects can assist in system design and in substantial improvements in video-conferencing technology such mitigating obstacles to communication in further iterations of as higher-resolution images and larger screens. Regardless of the the technology. Ensuring natural (or the best approximation causes for its growing popularity, like other technology-assisted possible) eye contact in video-conferencing can increase the communication media, video-conferencing reveals important as- effectiveness of this mode of communication. pects of human communication that should be considered in the development of new applications afforded by ever-advancing technology. 2. Eye contact in communication ⇑ Corresponding author. Tel.: +1 585 210 8513. Gaze can be defined as directed looking at any object, person, or E-mail addresses: [email protected] (L.S. Bohannon), esa.rantanen direction. Eye contact is gaze directed at another’s eyes, and @rit.edu (E.M. Rantanen). mutual eye contact occurs when two people make eye contact 1 Tel.: +1 585 475 4412. simultaneously. When two people are in casual conversation, eye 0141-9382/$ - see front matter Ó 2012 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.displa.2012.10.009 178 L.S. Bohannon et al. / Displays 34 (2013) 177–185 contact occurs about 61% of the time, about half of which is mutual in self-consciousness during those years of development, and in- eye contact. People make eye contact 75% of the time while creases again in adulthood. listening and 41% of the time while speaking. However, these figures may vary based on the context of the conversation. While 2.3. Attention and memory it seems clear that people spend more total time looking at the other while listening than speaking, the length of the gazes also Gaze can be a powerful director of attention. To illustrate this, vary. In an experiment with conversational dyads, participants one study presented participants wearing an eye-tracker with a made long gazes at their partner and looked away only briefly screen for 1500 ms displaying a fixation point between two when listening, but made shorter gazes of equal length both at squares. Then the fixation point changed color. The color and away from their conversational partner while speaking. was a cue to the participant to look either to the left or the right. Also, there is evidence that conversational partners tend to spend The fixation point disappeared after 150 ms, and then a photo- approximately the same amount of time looking at each other, graph of a person’s face appeared after either 50, 100, or 150 ms. adjusting their gaze based on the other’s gaze. The impact of The eyes of the face were looking either to the left, right, or center. gaze, and more specifically eye contact, on our social interactions The face stayed on the screen for 150 ms. In the trials with the as humans is significant. While we rely on a number of nonverbal 50 ms interval there were more antisaccades (saccades looking in cues to communicate information to others, eye contact seems to the opposite direction of the color cue) when the eye direction stand out distinctly from the rest. was incongruent with the color cue. That is, the gaze direction of the face overrode the color cue instructions. This is evidence 2.1. Biological and developmental considerations that our attention tends to follow the gaze of another. Similarly, participants were faster to respond to a target when it was accom- There is evidence that humans become sensitive to gaze direc- panied by a schematic face gazing in the same direction , pos- tion by around 5 months of age , and infants as soon as 48 h sibly indicating that the tendency to follow another’s gaze is after birth prefer faces that are gazing at them to faces that are reflexive in nature. Many illusions and card tricks work in part looking away. For example, infants looked significantly longer by misdirecting the gaze of viewers. Participants viewing a trick at photographs of faces whose eyes were looking straight ahead while being eye-tracked follow the gaze of the person performing over faces with averted eyes. Furthermore, calm 9–12-week-old in- the trick, thereby missing critical information that would reveal fants preferred a familiar face to a stranger’s face when the familiar the trick. The timing of gaze direction effects is consistent with face was previously associated with both eye contact and sucrose prioritized visual processing of this information. delivery. Memory, too, is affected by gaze direction. When presented The amygdala, a brain structure related to processing emotional with a prerecorded sales pitch for a soap product, participants stimuli, is thought to be a key neural component of gaze detection; remembered more about the product in the condition where the a patient with an amygdala lesion gazed more at people’s mouths salesman made eye contact with the camera (the participant) than and less at their eyes during conversation as compared to subjects when he made no eye contact. without an amygdala lesion. Other cortical regions related to visual processing have also been shown to respond differentially 2.4. Impression formation for gaze direction. Area V5 of the visual cortex (part of the dorsal visual pathway involved in motion perception and coding actions) Gaze direction can influence the likeability and attractiveness of is more active when viewing images of faces whose eyes are a person as perceived by another. When participants in a study animated to make eye contact as opposed to faces whose eyes [20,21] viewed female faces that made a gaze shift toward them, are animated away from eye contact. These results suggest they rated the faces to be more likeable than when the eyes of that eye contact is an integral (or ‘‘hard-wired’’) part of human the face made an animated shift away from the participant. Males interaction. rated female faces that made eye contact as more attractive than female faces whose eyes looked away. Couples that scored 2.2. Cultural, age, and sex differences higher on a questionnaire test measuring level of romantic love en- gaged in more eye contact than couples that had a lower romantic Eye contact during conversation varies from culture to culture, love score. however. Arabs, Latin Americans, and Southern Europeans make It also appears that believed eye contact, or what we are told more eye contact during conversation than Asians and Northern about the level of our own or another’s eye contact can affect our Europeans. The English prefer to stand farther away and have perception of that person. Kleinke et al. had male–female unwavering eye contact while being attentive to what another is dyads engage in a conversation. After the conversation, the exper- saying while Americans tend to stand closer and occasionally imenter told the participants that the amount of gaze of one of glance away from, and between, the two eyes of the person speak- them was either less, about the same as, or more than that of most ing. people. Partners alleged to have made less than average eye con- In Japanese culture eye contact is considered rude, and people tact were rated as less attentive. When males were told they gazed are taught to look at a person’s Adam’s apple instead of the eyes less than average at their female partner, they rated her more. The level of eye-contact avoidance in the Japanese culture favorably. Conversely, females who were told they gazed at their also depends on social rank – eye contact with a superior should male partner more than average rated him more positively. When most definitely be avoided. The different effects of these cul- males were told their female partner gazed an average amount, tural norms in communication are unknown. However, it is inter- they rated her as more relaxed. Conversely, when females were esting to note that the lack of eye contact in one culture can be just told their male partner gazed an average amount they rated him as important a component of communication as the presence of as less relaxed. Males who were told their female partners had less eye contact in another. than average gaze rated her as least attractive while females who In conversation, females make more eye contact while talking were told their male partner had more than average gaze rated than do males. The amount of gaze during conversation in- him least attractive. Both males and females who were told their creases from between the ranges 4–6 and 6–9 years of age. Gaze partner had more than average gaze rated him or her as being decreases around 10–12 years of age, perhaps due to an increase more sincere. Thus, regardless of the actual gaze of another L.S. Bohannon et al. / Displays 34 (2013) 177–185 179 person, the amount of gaze we think someone is directing at us af- Gamer and Hecht coined the term ‘‘cone of gaze’’ to refer to fects our impression of him or her. the ‘‘range of considerable width wherein a person feels looked at’’ (p. 706). 2.5. Other impacts of eye contact Many studies examining gaze perception used an experimental setup involving pairs of people, one called the looker and the other Eye contact also plays an important role in other social interac- called the observer [27–32] (see Table 1 for a summary). The looker tions. For example, the presence or absence of eye contact can af- is a confederate to the experiment and looks at different points on fect the perceived emotion expressed by the face. When presented a scale invisible to the observer. The observer then makes some with photographs of neutral faces, half of which gazed straight type of judgment about where the looker is looking. These gaze ahead (made eye contact) and half of which had averted gaze, par- detection studies used different metrics for measuring the just ticipants were more likely to judge neutral faces that made eye noticeable deviation in gaze; some studies use degrees of visual an- contact as expressing anger or joy, whereas neutral faces with gle and others use centimeters. When converted to degrees of vi- averted gaze were more likely to be rated fearful or sad. When sual angle the just noticeable deviation in gaze varies between participants were presented with composite faces expressing both 0.73° and 9.3°. This large variation between results is not surpris- fear and anger, if the gaze was averted the face was more likely to ing given the methodological differences in the studies. Thus, it be rated as fearful. When the composite faces were gazing at the is perhaps not useful to compare results directly to one another participant, the face was equally likely to be rated as either fear but instead to look for patterns; the important information to take or anger. Anger was rated as more intense when eye contact was from research on gaze direction perception is that (1) people are present and fear was rated as more intense when gaze was averted very sensitive to deviations in gaze from eye contact (2) except,. perhaps to downward gaze and (3) perception of gaze direction In one unique experiment, an ‘‘invader’’ either stared or did not is influenced by head orientation. All three of these points are rel- stare as he tried to ‘‘invade’’ the pew of local church-goers. If evant to video-conferencing. the invader stared, the worshipper was likely to slide down the pew and make room for the person. However, in infrequent cases 3.2. Gaze perception during video-conferencing when the worshipper stared back at the invader, he or she typically did not make room for the invader. If the invader did not stare, the Some studies on gaze perception focus specifically on eye con- worshipper was less likely to move. It was uncertain if staring in tact perception and video-conferencing. These studies explore the this instance had to do with communicating aggression, or if its ab- role of video and parallax on gaze perception. sence maintained a certain anonymity that made it easier for Chen setup a ‘‘recording studio’’ in which a camera was church-goers to ignore the request for a seat , but in any case placed behind a hole in the middle of a projector screen. Points underscores the importance of eye contact as nonverbal were drawn 1° apart on the screen along a horizontal, vertical, communication. and diagonal orientations indicating targets for the looker. The Eye-contact plays a role in perceived trust. US Customs inspec- downward direction covered 15° and the other seven lines of tors were more likely to want to search a passenger in a mock points covered a range of 5° each to form a star-like pattern. The interview when the passenger diverted his gaze frequently. observer looked at videos of the looker gazing at the different This was true despite the fact that gaze avoidance was not corre- points in random order. It was the observer’s task to indicate lated with lying [25,26]. Eye contact impacts other social interac- whether the looker was or was not looking at him. tions such as impression formation, emotion and compliance and Observers were very sensitive to eye contact when the these affects will be examined later in this paper in the context looker gazed up, left, or to the right but less sensitive when the of video-conferencing. looker gazed below the screen as shown in Table 1. When the looker was looking down, observers were much less sensitive to eye contact; the looker could look as far as 10° away before 3. Eye contact in video-conferencing the observer reported eye contact was lost. One key difference between Chen’s study and a video-conferencing system is that Feeling as if the person with whom you are conversing is look- participants in Chen’s study made judgments about images of faces ing at your eyes is important; when given the option between that were taken looking directly into the camera as opposed to using a video-conferencing system that enables eye contact and from an elevated angle as is the case with video conferencing. one that does not, people overwhelmingly prefer the system that Chen suggested that horizontal eye contact sensitivity to enables eye contact. The geometry of video-conferencing influ- gaze deviation might be accurate because, as the eye rotates hori- ences the perception of eye contact. Current video-conferencing zontally, the sclera becomes more visible. But as the eye rotates technology is designed such that a camera sits atop a screen. The downward it becomes difficult to know if a person is making direct conversation partner’s image appears on the screen and the camera eye contact because less or no sclera can be seen. Chen sug- captures one’s face (and vice versa). For each conversation partner gests we perceive downward gaze as great as 10° to be eye contact it appears as if the other person is looking downward – but each and recommends that in a video-conferencing system the eyes dis- partner is looking right at his or her partner’s eyes on the screen. played on the screen should be located approximately 5° below the This discrepancy between the camera’s view of the head and where camera to create the illusion of eye contact. the person is gazing is a form of parallax. One would need to look To examine gaze direction perception under video-mediated directly into the camera for there to be eye contact. This is not conditions, Gale and Monk had a looker and observer either what people do, because the intention is to look at the conversa- sit across from each other with the targets between them, or a vi- tion partner’s face, not the camera. deo camera was placed where the observer’s head would have been. In both conditions both the looker and the observer were 3.1. Cone of gaze seated at eye-level with one another. Targets were marked on a vertical line that extended upward to the observer and on a hori- Early research on gaze perception focused on finding the zontal arc such that every point on the arc was the same distance threshold for what is considered eye contact. That is, how far can from the looker. Targets were marked such that the looker had to gaze deviate from the eyes and still be considered eye contact? rotate his or her head 2° to look from one to the other. In one 180 Table 1 A summary of studies examining gaze perception sensitivity. Study Apparatus and stimuli Angles Distance Task Gaze sensitivity Gibson and Pick Targets marked on wall behind O A 6 cm horizontal span with targets 200 cm O responded ‘‘yes’’ or ‘‘no’’ to the question of The just noticeable deviation was 2.58° or (1963) separated by 2.9° whether or not L was looking directly at him about the distance from O’s nose to the edge of her face Cline (1967) Transparent target board through Targets along a vertical and 122 cm O pointed to where she thought L was looking. The threshold at which O still perceived L to which O viewed a mirror which horizontal axis separated by 2° She also said when she thought L was looking be looking at him was: reflected L at her 0.73° horizontal and 1.27° vertical for experiment 1 1.34° horizontal for experiment 2 and 2.48° when O’s head was rotated 30° right or left Anstis et al. (1969) Horizontal scale placed between L Targets along a horizontal axis 84 cm O indicated on a scale where L was looking by Overestimated where L was looking by and O 6 cm above eye level. Scale spanning 0°, 5°, 10°, 15°, 20° to the pointing with a rod between 50% and 87% rotated for experiment 2 left and right When L’s head rotated 30° left of O, O estimated gaze to be 7.7° to the right. When rotated right, estimation was 5.44° to the left When TV was rotated to O’s left, estimation was 1.81° to the left. When rotated right then 4.56° to the right When scale was vertical, estimates were true to scale Chen (2002) L’s gaze positions were recorded by a Targets were separated by 1° 480 cm (O and L O indicated if L was looking at her or not When L gazed up, down, left, or right, L camera through a hole in a projector increments downward to 15°. There each sat 240 cm could look at most 1° away from camera before screen which had targets drawn on it. were seven other lines radiating from from camera/ O perceived L was no longer looking and her L was prerecorded on video tape the center that spanned 5° each and screen) formed a star-like pattern L could look 10° downward from the center L.S. Bohannon et al. / Displays 34 (2013) 177–185 of the camera before O noticed L was no longer looking at her Gamer and Hecht Moveable eyes in a virtual and real Eyes of both the virtual and real head 100 cm and O moved L’s eyes either toward or away from The cone of gaze was 9.3° at a distance of (2007) looker eyes moved 1° at a time with the aid of 500 cm him until he thought L was looking directly at 100 cm and 8.2° at a distance of 500 cm for the computer software or an electronic him virtual head pointer device, respectively. Eyes could move as far as 10° to the right or left The cone was 8.2° at 100 cm and 3.9° at 500 cm for the real head When L’s head was rotated to right or left 10° the gaze was perceived to be shifted more in the direction of the head rotation than it actually was. The effect was greater for the 100 cm distance than the 500 cm distance L.S. Bohannon et al. / Displays 34 (2013) 177–185 181 condition, observers were allowed to see the eye and head move- to study communication because the map task requires informa- ments of the looker and in other conditions they were not. The loo- tion transfer and reception through spoken dialogue. In the map ker first looked at the extreme points and center on the scale to task, each participant is provided a map with landmarks scattered ‘‘calibrate’’ the observer. The observer then made estimations about it. The two maps contain some similar landmarks but are about where the looker was looking. slightly different. One person is designated to be the instruction gi- The ability to see eye and head movements did not affect gaze ver, and the other person is designated to be the instruction recei- direction perception ability. Gaze estimation was accurate to with- ver. The instruction giver’s map contains a trail. The goal of the task in about a target point (2°) for both live and video-mediated esti- is for the giver to verbally articulate the trail’s path to the receiver mating conditions. Similar to Chen , gaze estimation error who must replicate the trail on his own map. Error in the final re- was smaller for horizontal than for vertical directions. In a second sult can then be measured and provide one measure of success. experiment the calibration was eliminated and observers made Boyle et al. evaluated performance and communication estimations about directions for both horizontal and vertical – gaze efficacy on the map task in two conditions: co-present audio-only direction perception was also very accurate for the combination of and face-to-face. Performance was measured by placing the directions. instruction giver’s map on top of the follower’s map and calculat- Grayson and Monk evaluated the perception of horizontal ing the difference in the paths (deviation) in square centimeters. gaze direction with desktop video-conferencing systems. In one While performance did not differ between the two conditions, condition the camera was placed directly above the image of the communication patterns did. looker and in the other it was offset. Additionally, the size of the Conversation efficiency was measured by number of turns, looker’s face on the screen was varied. The image was either large number of words, number of words per turn, number of interrup- (352 288 pixels) or medium (176 144 pixels). The looker ran- tions (and interruption rate), and number of back channel re- domly looked (moving both eyes and head) at one of six points sponses (and back channel rate). Conversation pairs who could (43 mm apart) along a horizontal line that extended from either not see each other took more turns and used more words than side of the image of the person on the screen. The observer esti- those who could see each other. Participants who could not see mated which point the looker was gazing at and the looker told each other took shorter turns, that is, they spoke fewer words the observer what the correct answer was. per turn, than did participants who could see each other. Partici- Image size had no effect on the perception of gaze direction pants who could not see each other while completing the map. When the camera was positioned over the face of the partic- mask interrupted each other twice as often as participants who ipant, gaze estimation was quite accurate (84%) but when the cam- could see each other. Conversations in which participants could era was not place directly above the person being looked at, gaze not see each other were longer (as measured in number of words) estimation dropped (67% accuracy). When the camera is positioned than conversations in which participants could not see each other, directly over the image of the face, gaze direction perception accu- and the interruption rate (percentage of turns containing interrup- racy decreases the further the gaze deviates from the centered tions was higher for the no-visibility condition. A back channel re- ‘‘looking at me’’ position. Estimations showed little variation when sponse was defined as ‘‘a turn which consisted of an ‘uhuh’ or the camera was offset. ‘mhm’’’ (, p. 10). Back channel responses were greater in both In a second experiment, Grayson and Monk examined gaze number and rate for conversations that enabled visibility than con- perception with a medium (176 144 pixels) and a small (88 72 versations that did not. Thus, the end product (the path) did not pixels) image of the person on the screen. Also, in half of the trials differ as a function of eye contact, but conversational efficiency the looker held a card with the fixation point on it over the image was increased when the pair could see each other. of the observer whenever he made a fixation – this was to elimi- Boyle et al. also examined the instruction follower’s gaze nate the possibility that subtle communication could reveal gaze toward the instruction giver. Conversations were videotaped and location when the looker was gazing at the image of the observer. the transcribed dialogue was marked every time it was estimated Accurate detection of the eye contact position was 87% for the the follower looked up and at the giver. When discussing a feature medium image and 68% for the small picture. Discrimination be- that differed between the two map versions, the instruction fol- tween points decreased for points further from the center. Making lower gazed at the giver more often than when discussing a feature the observer invisible to the looker did not influence judgments of that was the same. Other research examining gaze and turn taking gaze direction. found that people tend to look more at their conversational partner Unlike most of the previous gaze research detailed earlier, these when they are ready to stop talking and let their partner begin two studies [33,34] did not focus on thresholds of when perceived speaking. Participants in Boyle et al.’s experiment may eye contact was lost. Instead, they focused on estimation of where have looked at their partner more when discussing incongruent someone was looking. When there is no parallax, estimation of map features because they were trying to elicit verbal explanation gaze direction made from video seems to be quite accurate from their partner regarding more details about the features. but once parallax is introduced as in desktop video-conferencing, According to Boyle et al.’s measures of conversation effi- gaze direction perception accuracy decreases as the gaze direction ciency, the ability to view one’s conversation partner significantly deviates from a centered position, directly under the camera. increases the amount of information being conveyed in a shorter The varying effects of image size are consistent with the idea that amount of time and with fewer words. It can be inferred that inter- small images lead to a poor user experience, but that camera posi- ruptions result from some sort of confusion about either whose turn tion matters more than image size. it is or about the task. Back channel responses can be interpreted as a means of communicating understanding and agreement. 4. Effects of eye contact on video-mediated communication Doherty-Sneddon et al. distinguished between the surface features and pragmatic function of dialogue by exploring dialogue 4.1. Communication efficacy and gaze under the context of five communication media – co- present audio, remote audio, face-to-face, video-mediated that Some researchers have used the map task as a tool to analyzing enabled eye contact and video-mediated that did not enable eye the effects of video-conferencing and eye contact on communica- contact. Like Boyle et al. , Doherty-Sneddon et al. used tion. The map task was originally suggested by Anderson et al. the map mask as a cooperative task for their participants. as a way to teach and evaluate language. It is a useful tool Doherty-Sneddon et al. analyzed the conversations between 182 L.S. Bohannon et al. / Displays 34 (2013) 177–185 participants by use of Conversational Games Analysis. The Conver- significantly more in the video-conferencing condition that sational Games Analysis technique, developed by Kowtko et al. enabled eye contact than in the video-conferencing condition that , breaks the content of conversation into games that may con- did not enable eye contact. They also gazed more in the video- tain several moves. Portions of conversations are given the analogy conferencing condition that enabled eye contact than in the of a game because there are ‘‘rules’’ or conventions that are known face-to-face condition of the first experiment. by both parties. In their first experiment, Doherty-Sneddon et al. compared 4.2. Trust the dialogue of participants who did the paper and pencil map task under audio-only conditions to those who did the task face-to-face. Besides establishing mutual knowledge, other important social In the audio-only condition, participant’s dialogues contained 16% interactions take place during communication and can be im- more conversational games than in the face-to-face condition. pacted by video-conferencing. Building trust, impression formation When participants could not see each other they had a greater need and establishing social status are all affected by different forms of to confirm and check their understanding of one another; audio- mediated communication. only conditions contained significantly more conversational games To determine the effect of eye contact in video-mediated com- that checked and confirmed information. Videos of the face-to-face munication on trust, Bekkering created a scenario in which conversation were coded for instances of gaze between partici- participants indicated the trustworthiness of a message delivered pants. Doherty-Sneddon et al. considered gaze to have by people in three video-conferencing conditions. Participants occurred when they perceived one partner had looked at the face were to imagine a scenario in which they had just been denied a of the other. Participants tended to look at the other’s face at points job. The person at human resources informed them that the reason in the conversation when they were verbally eliciting feedback. they did not get the job might have had something to do with a Performance on the map task did not differ between conditions. problem with letters of references. The participants were to imag- In their second experiment, Doherty-Sneddon et al. com- ine that they called the two people they had asked to write letters pared dialogue structure and gaze between remote-audio (accom- of references to check if letters had been submitted. Participants plished using microphones and speakers) and video-mediated then listened to messages from the two people - one familiar per- communication that either enabled (by use of a video-tunnel, son (their professor) and one unfamiliar person (a minister from a which uses mirrors to create the illusion of eye-contact) or pre- neighboring town) confirming that they did indeed send letters. vented eye contact. Eye contact was prevented in one condition The messages came in the form of an email, a voicemail, a video by raising the camera so that the participants were ‘‘now looking in which the person was making eye contact (the camera was slightly down on his or her partner’’ (, p. 116). In contrast to eye level with the face), a video in which the camera was placed Boyle et al. who found that audio-only conversation required above the person’s face, and a video in which the camera was more words and turns, Doherty-Sneddon et al. reported that placed to the side of the person’s face. Each participant listened there was no difference in number of words or turns between re- to the messages in all their forms (the script was the same for each) mote audio-only conversation and video-mediated conversation and indicated how much they trusted that the person had sent in that did not enable eye contact. In fact, video-mediated conversa- their letter of reference. tion that enabled eye contact had significantly more words and Whether or not the person delivering the message was familiar turns than conversations in the no eye contact and audio-only to the participant did not affect perceived trust. When compared to conditions. Doherty-Sneddon et al. explained this unexpected the video that enabled eye contact (center camera), videos that result by pointing out that the video-tunnel allowing eye contact prevented eye contact (side and top camera) resulted in lower may have been a novel experience for participants that fascinated perceived trust scores. Trust scores did not differ between them, resulting in more conversation. Another possibility is that voicemail and centered video. Trust was significantly better for the greater number of words recorded in the eye-contact condition the centered video than for email. In summary, video- could be explained by an increase in attention required by looking communication that prevents eye contact can reduce trust when at another’s face; some research suggests that increased eye compared to video-communication that does. Voicemail enabled contact during conversation increases the number of spoken just as much trust as the video that created eye contact, perhaps hesitations. Repetitions, fillers like ‘‘er’’ and ‘‘um,’’ false starts, because lack of eye contact cannot be perceived in audio-only and other remarks that may be irrelevant to semantic communica- communication. tion such as ‘‘I mean’’ and ‘‘sort of’’ were almost double when When participating in games that require group cooperation, participants looked constantly at an interviewer as compared to participants trust each other more when communicating face- when they were allowed to avert their gaze. In both the to-face as compared to computer-mediated media. Partici- video-mediated conditions, instruction givers initiated confirmed pants, who played the Daytrader social dilemma game in groups information significantly less than instruction givers in the of three, across four media conditions (face-to-face, video-confer- audio-only condition. However, when video-conferencing was encing, three-way phone conference, and text chat), had to make compared to remote audio-only conversations, there was no an investment choice that was either individual in which they were reduction of the number times instruction followers checked infor- guaranteed two tokens for themselves or as a group in which all mation. It appears as if, like face-to-face communication, video- players were guaranteed three tokens so long as they all invested. mediated communication increases feedback to instruction givers Every five rounds of the game a bonus of 90 tokens was given to when compared to audio-only communication. However, unlike whoever had the most tokens at that point, and participants had face-to-face communication (in experiment 1), video-mediated the opportunity to discuss and agree upon a strategy for the next communication does not reduce the need for instruction followers five rounds. If, for the five rounds preceding the bonus, all players to check their knowledge of the instructions when compared to cooperated and invested as a group, the bonus was split three ways. audio-only conversations. These findings suggest that, while Participants who played the game in the text chat condition video-mediated communication is an improvement over audio- made significantly fewer tokens from their investments than did only communication, it is not comparable to face-to-face commu- groups in the other three conditions. Group investments started nication in its ability to facilitate efficient communication. high and remained high for the face-to-face condition. As Doherty-Sneddon et al. analyzed gaze between the the game progressed, group investments increased for the audio conditions and found that participants looked at each other and video conditions. Group investments started low and stayed L.S. Bohannon et al. / Displays 34 (2013) 177–185 183 low for the text chat condition. If the amount of group investment persuaded by a confederate to incorrectly rank order survival is interpreted as trust, establishing trust was delayed in the audio equipment when they were communicating over a video-confer- and video conditions but gradually reached the same level as in encing setup that did not facilitate selective gaze compared to a face-to-face condition. Sometimes after each discussion, group video-conferencing setup that did enable selective gaze and via members would gradually deviate from group cooperation. Coop- face-to-face communication. Furthermore, when performing eration was usually strongest right after a discussion. This measure a murder mystery-solving task, groups of participants were more of defection was used as an indicator of the fragility of trust. likely to communicate unshared information over video- Defection for the computer-mediated conditions was significantly conferencing than face-to-face. greater than for the face-to-face condition. Bos et al. inter- A number of factors are likely responsible for the impact of preted these results to mean that trust is both more delayed and video-conferencing on impression formation over face-to-face more fragile in computer-mediated-communication modes than interactions. The quality and speed of the video and audio signal in face-to-face communication. can mitigate the transmission of subtle non-verbal communication Also using the Daytrader dilemma, Nguyen and Canny. Furthermore, people may be more self-conscious when talk- examined trust under three media conditions: face-to-face, ing over video-conferencing (intensified by being able to see their non-directional video-conferencing, and directional video- own image on camera), and may therefore behave differently than conferencing. Directional video-conferencing was defined as they normally do. Additionally, eye contact perception is not video-conferencing that enables accurate cues about gaze and accurate in video-conferencing, and the presence or absence of gesture direction and was achieved by Nguyen and Canny’s eye contact can affect the impression of personality and emotions MultiView video-conferencing system. This system differs from. traditional video-conferencing systems in that a camera is placed in front of each member of the group giving them an accurate per- spective of the other group members they are viewing on the 5. Solutions to parallax in video-conferencing screen. The amount of cooperative investments made by groups did not The role of eye contact in human communication and the differ between the face-to-face condition and the directional impact of the now ‘‘traditional’’ video-conferencing arrangement video-conferencing condition, and both of these were significantly on perceived eye contact underline the importance of creating higher than for non-directional video-conferencing. There was no the optimal perception of eye contact during video-conferencing. difference in delay of trust between the three conditions. However, Solutions to the lack of eye contact in video-conferencing are trust was significantly more fragile in the non-directional varied and involve hardware and software approaches. Perhaps video-conferencing system than in both the directional video- one of the earliest solutions was the design of the video-tunnel conferencing system and the face-to-face conditions. in Doherty-Sneddon, et al.. This comprises a half-silvered Additionally, participants’ self-reported trust was significantly mirror placed at a 45° angle from a screen that is lying flat on lower in the non-directional video-conferencing system than in the surface in front of the person communicating. The image both the directional video-conferencing system and the face- of the screen (and thus the other person’s face) is reflected onto to-face conditions. The results of this study indicate that a the mirror. A camera (or in some cases multiple cameras) is placed video-conferencing system that affords more eye contact than behind the mirror so that the person is both looking at the image of the traditional video-conferencing system will create group trust the eyes of the other and at the same time looking directly into the levels similar to those seen in face-to-face group meetings. camera. This creates the appearance of eye contact. Such a setup can be limiting in terms of screen size and sometimes the person 4.3. Impression formation is required to place his head in a chin rest to maintain eye contact. A modern-day attempt to adapt the video-tunnel to today’s In a study particular to video-conferencing, Fullwood used a existing technology involves a miniature video tunnel that fits over unique interaction task to examine the role of communication med- small monitor-mounted web cameras. ia on impression formation. Same-sex participant pairs who were Similar in concept to the video-tunnel is embedding a camera in strangers to each other communicated either face-to-face or via the display. Tapia et al. embedded a camera (1.5 mm in diam- video-conferencing. One partner in each pair was the ‘‘thought eter) onto a surface and projected the other’s face onto that surface. transmitter’’ and the other was the ‘‘mind-reader.’’ The thought This enables the speaker to look at both the image of the other transmitter was given a card with a numbered list of 10 colors on person and the camera, creating eye contact. This is not a viable solu- it. The mind-reader was shown at list for 10 s that contained all pos- tion for current laptops and monitors in commercial production. sible colors he would be exposed to. The thought transmitters had Ott et al. used a combination of multiple cameras and to attempt to transmit each color to the mind-reader in the order software to create eye contact. A camera was placed both above that they appeared on the page by thinking about each color for and below the screen; the top camera tilted slightly downward 5 s. After each 5 s, the mind-reader had to guess what color the and the bottom camera tilted slightly upward. Together these transmitter had been thinking of. The mind reader was not given cameras captured the same person from two different viewpoints. feedback as to whether they had identified the colors correctly. Corresponding pixels were merged and a combination image was The participants in the pair then swapped roles and afterwards used to create a single image in which it appeared the person filled out questionnaires about their partners which assessed their was looking straight into the camera. However, it was not possible feelings about how much they liked the other participant, how to capture the entire face resulting in some missing pieces in the intelligent they thought he or she was, and how good at reading final image. minds they thought he or she was. The participants in the While many alternate video-conferencing systems have been face-to-face condition liked their partners more than in the video- developed, the most prevalent video-mediated communication conferencing condition and rated them as more intelligent. system remains the version where one camera is placed above When groups of participants had discussions about a common the screen. Such systems are cost-efficient, portable, and compati- strategy to survive on the moon (the Lost on the Moon Task) ble with many types of computer systems. Most laptops and conditions enabling selective gaze had greater effects on behavior tablets are now made with a small camera placed directly above than other conditions. Participants were more easily the center of the display. Solutions that aim to improve eye contact 184 L.S. Bohannon et al. / Displays 34 (2013) 177–185 perception under the constraints of this prevalent video- References conferencing system are thereby most likely to be adopted by the majority of current users. AT&T, 1970: Picturephone, 2010. (retrieved 28.10.10). One such solution is software that alters the face image so as to L. Wolgemuth, Knights of the virtual table: video conferencing grows up create the illusion of eye contact. The software replaces the thanks to ‘telepresence’ systems, US News and World Report 144 (2008) 48– eyes with computer-generated eyes and adjusts the head pose by 50. R.L. Daft, R.H. Lengel, Organizational information requirements, media mapping the face image onto a head model. The software is able richness, and structural design, Management Science 32 (1986) 554–571. to detect where the person is looking and adjust the appearance N. Guéguen, C. Jacob, Social presence reinforcement and computer-mediated of the gaze accordingly. The disadvantage of the software is that communication: the effect of the solicitor’s photography on compliance to a survey request made by e-mail, CyberPyschology and Behavior 5 (2002) 139– head movements of more than 30° cause distortion. 142. However, simply by understanding human perception of gaze S.H. McNelley. The Significance of Eye Contact While Desktop Video during video-conferencing, as was summarized in Section 4 of this Conferencing, Ph.D. Dissertation, United States International University, paper, much simpler solutions are possible. One algorithm takes United States – California, 2001. M. Argyle, M. Cook, Gaze and Mutual Gaze, Cambridge University Press, advantage of the large margin of error when perceiving gaze in a London, 1976. downward direction by locating the eyes of conversational part- A. Kendon, Some functions of gaze-direction in social interaction, Acta ners and always displaying the image of the eyes close to the top Psychologica 26 (1967) 22–63. A.J. Caron, R. Caron, J. Roberts, R. Brooks, Infant sensitivity to deviations in of the display where the camera is located. Without the use dynamic facial–vocal displays: the role of eye regard, Developmental of extra equipment or computer programs, the average user can Psychology 33 (1997) 802–813. improve his or her video-conferencing experience by ensuring that T. Farroni, E. Menon, M.H. Johnson, Factors influencing newborns’ preference for faces with eye contact, Journal of Experimental Child Psychology 95 (2006) his or her own image is close to the top of the camera’s field of 298–308. view, the image of the conversational partner is as close to the E.M. Blass, C.A. Camp, The ontogeny of face recognition: eye contact and sweet camera as possible, and by sitting back from the display until the taste induce face preference in 9- and 12-week-old human infants, Developmental Psychology 37 (2001) 762–774. perception of eye contact is achieved. M.L. Spezio, P.S. Huang, F. Castelli, R. Adolphs, Amygdala damage impairs eye contact during conversations with real people, The Journal of Neuroscience 27 (2007) 3994–3997. 6. Conclusion S. Watanabe, R. Kakigi, K. Miki, A. Puce, Human MT/V5 activity on viewing eye gaze changes in others: a magnetoencephalographic study, Brain Research Much of the literature on gaze direction sensitivity emphasizes 1092 (2006) 152–160. E.T. Hall, The Hidden Dimension, Anchor Books Doubleday & Company, Inc., differences among samples (based on gender, culture, role, etc.). Garden City, NY, 1969. Perhaps the similarities should be emphasized. In general, allowing J.C. Nixon, J.F. West, Intercultural preparation for managers going to Japan, eye contact makes communication richer and more efficient at the Mid-American Journal of Business 10 (1995) 57–64. M.H. Levine, B. Sutton-Smith, Effects of age, sex, and task on visual behavior same time. Trust is improved, with clear implications for how during dyadic interaction, Developmental Psychology 9 (1973) 400–405. business transactions may improve with eye contact facilitated. P. Ricciardelli, E. Bricolo, S.M. Aglioti, L.C. Chelazzi, My eyes want to look where There are very expensive solutions available for ‘life-like’ video- your eyes are looking: exploring the tendency to imitate another individual’s gaze, NeuroReport 13 (2002) 2259–2264. conferencing, but the vast majority of users are employing Skype C. Friesen, C. Moore, A. Kingston, Does gaze direction really trigger a reflexive or analogous free solutions on their existing computer systems. shift of spatial attention?, Brain and Cognition 57 (2004) 66–69 Based on the review of the literature, it seems clear that users G. Kuhn, B.W. Tatler, G.G. Cole, You look where I look! Effect of gaze cues on overt and covert attention in misdirection, Visual Cognition 17 (2009) 925– can improve outcomes with a few simple measures: (1) by keeping 944. the interlocutors’ image as near the camera as possible; (2) by C. Fullwood, G. Doherty-Sneddon, Effect of gazing at the camera during a video sitting back a bit to improve the impression of mutual eye contact; link on recall, Applied Ergonomics 37 (2006) 167–175. M. Mason, E. Tatkow, C. Macrae, The look of love: gaze shifts and person and, (3) smaller views of interlocutors should be avoided. The perception, Psychological Science 16 (2005) 236–239. ordering of these three measures reflects their relative impact from Z. Rubin, Measurement of romantic love, Journal of Personality and Social most to least important given the literature reviewed. Video- Psychology 16 (1970) 265–273. conferencing software that facilitates such adjustments may C.L. Kleinke, A.A. Bustos, F.B. Meeker, Effects of self-attributed and other- attributed gaze on interpersonal evaluations between males a females, Journal provide better user experience. Furthermore, users can make com- of Experimental Social Psychology 9 (1973) 154–163. munication via video-conferencing more like talking in-person by R.B. Adams, R.E. Kleck, Effects of direct and averted gaze on the perception of implementing some other simple methods - making sure hands facially communicated emotion, Emotion 5 (2005) 3–11. D.E. Campbell, G.E. Lancioni, The effects of staring and pew invasion in church and arms are in view of the camera assure that more non-verbal settings, Journal of Social Psychology 108 (1979) 19–24. cues are communicated, and initially making face-to-face contact, R.E. Kraut, D. Poe, Behavioral roots of person perception: the deception of or engaging in an ice-breaker activity builds report between judgments of customs inspectors and laymen, Journal of Personality and Social Psychology 39 (1980) 784–798. communicants. Communication. Merriam-Webster.com. (n.d.). (retrieved 02.08.12). verbal and non-verbal communication, also affecting social M. Gamer, H. Hecht, Are you looking at me? Measuring the cone of gaze, Journal of Experimental Psychology 33 (2007) 705–715. onstructs such as trust and impression formation. As video- S.R.H. Langton, H. Honeyman, E. Tessler, The influence of head contour and conferencing becomes a more pervasive form of sharing informa- nose angle on the perception of eye-gaze direction, Perception and tion and relationships, technology needs to evolve to approach Psychophysics 66 (2004) 752–771. J.J. Gibson, A. Pick, Perception of another person’s looking behavior, American the effectiveness of face-to-face communication. Research on Journal of Psychology 76 (1963) 386–394. human sensitivity to gaze perception suggests the mounting of a M.G. Cline, The perception of where a person is looking, American Journal of camera above the screen is a good solution. Users appear to adapt Psychology 80 (1967) 41–51. to this viewing arrangement rapidly, but there is much greater S.M. Anstis, J.W. Mayhew, T. Morely, The perception of where a face or television ‘‘portrait’’ is looking, American Journal of Psychology 82 (1969) satisfaction with systems such as directional video-conferencing 474–489. or technical solutions such as the video tunnel or mounting a M. Chen, Leveraging the asymmetric sensitivity of eye contact for camera in the screen. Designers of video-conferencing systems videoconference, in: L. Terveen (Ed.), Proceedings of the AMC CHI 2002 Conference on Human Factors in Computing Systems, ACM Press, New York, need to be sensitive to the limitations of current systems, and 2002, pp. 49–56. research on gaze detection and eye contact needs to keep pace with C. Gale, A.F. Monk, Where am I looking? The accuracy of video-mediated gaze technological innovations. awareness, Perception and Psychophysics 62 (2000) 586–595. L.S. Bohannon et al. / Displays 34 (2013) 177–185 185 D.M. Grayson, A.F. Monk, Are you looking at me? Eye contact and desktop C. Fullwood, The effect of mediation on impression formation: a comparison of video conferencing, ACM Transactions on Computer–Human Interaction 10 face-to-face and video-mediated conditions, Applied Ergonomics 38 (2007) (2003) 221–242. 267–273. A. Anderson, G. Brown, R. Shillcock, G. Yule, Teaching Talk; Strategies P.J. Werkhoven, J.M. Schraagen, P.A.J. Punte, Seeing is believing: for Production and Assessment, Cambridge University Press, New York, communication performance under isotropic teleconferencing conditions, 1984. Displays 22 (2001) 37–149. E.A. Boyle, A.H. Anderson, A. Newlands, The effects of visibility on dialogue and C. Fullwood, M. Finn, Video-mediated communication and impression performance in a cooperative problem solving task, Language and Speech 37 formation: an integrative review, in: Adam C. Rayler (Ed.), Videoconferencing: (1994) 1–20. Technology, Impact and Applications, Nova Science, 2010, pp. 35–55. G. Doherty-Sneddon, A. Anderson, C. O’Malley, S. Langton, S. Garrond, V. Bruce, R. Vertegaal, I. Weevers, C. Sohn, C. Cheung, GAZE-2: Conveying Eye Contact in Face-to-face and video-mediated communication: a comparison of dialogue Group Video Conferencing Using Eye-Controlled Camera Direction, CHI 2003, structure and task performance, Journal of Experimental Psychology: Applied ACM Press, 2003, pp. 521–528. 3 (1997) 105–125. See Eye2Eye: Teleprompter for Web Cameras, 2006.. Research Paper Published on the Human Communication Research Center, E.M. Tapia, S.S. Intille, J. Rebula, S. Stoddard, Concept and Partial Prototype 1993.. Video: Ubiquitous Video Communication with the Perception of Eye Contact, G.W. Beattie, A further investigation of the cognitive interference hypothesis of Ubicomp 2003 Video, Program, 2003, 2 p. (un-numbered). gaze patterns during conversation, British Journal of Social Psychology 20 M. Ott, J.P. Lewis, I. Cox, Teleconferencing eye contract using a virtual camera, (1981) 243–248. in: S. Ashlund, K. Mullet, A. Henderson, E. Hollnagel, T. White (Eds.), INTERACT J.E. Bekkering, Visual angle in videoconferencing: the issue of trust, Ph.D. ‘93 and CHI ‘93 Conference Companion on Human Factors in Computing dissertation, Mississippi State University, United States – Mississippi, 2004. Systems (Amsterdam, The Netherlands, April 24-29, 1993), CHI ’93, ACM, New N. Bos, J. Olson, D. Gergle, G. Olson, Z. Wright, Effects of four computer- York, NY, 1993, pp. 109–110. mediated communications channels on trust development, in: Proceedings of J. Gemmell, K. Toyama, C.L. Zitnick, T. Kang, S. Seitz, Gaze Awareness for Video- CHI 2002, ACM Press, New York, 2002, pp. 135–140. Conferencing: A Software Approach. IEEE MultiMedia 7, 4 (October, 2000), D. Nguyen, J. Canny, MultiView: improving trust in group video conferencing 2000, pp. 26–35. through spatial faithfulness, in: CHI 2007 Proceedings (Trust & Engagement), J.W. McGowen, US Patent No. US8203595. Washington DC: US Patent and 2007, pp. 1465–1474. Trademark Office, 2012..