Name: loneliness assuaged: eye-tracking an audience watching barrage videos
Uploaded: 2020-05-29T13:30:00.000+00:00
Duration: 6 min 45 s
Description: Watch this Scientific Journal Video about Loneliness Assuaged: Eye-Tracking an Audience Watching Barrage Videos at JoVE.com

This study was supported by the National Social Science Fund of China for the project (19ZDA332) entitled &#8220;Deep Convergence of Media and Innovation Model of Social Governance in the New Era;&#8221; a PhD Start-up Fund of the Natural Science Foundation of Guangdong Province, China (2017A030310536); and the Fundamental Research Funds for Central Universities (19JNQM04). The authors would like to thank Dean/Professor Zhi Tingrong for support and investment in the eye tracking laboratory.

This protocol adheres to the Jinan University research guidelines. As only the medical school of the university has an IRB board, no other discipline is required to have IRB approvals. However, the researchers confirm that all ethical rules and regulations were followed. The project did not pose any physical or psychological harm to participants.
1. Participant screening for the experiment
<ol>
	<li>Recruit native Chinese speakers from a southern Chinese university with normal or corrected-to-normal vision and no hearing impairments nor psychiatric history.</li>
	<li>Measure loneliness using the third edition of the UCLA scale60,61. Categorize those who score 44 or above in the scale as the lonely group and place the rest in the normal group. In terms of gender and age distribution, the group corresponds to the demographic characteristics of barrage viewers47.</li>
</ol>
2. Stimuli construction
<ol>
	<li>Use rational appeal ads and emotional appeal ads as video stimuli. Positive emotional ads provide emotional support for lonely audiences, where rational ads do not.
	<ol>
		<li>Choose ads that are standalone video pieces that do not require contextual information to comprehend. The typical ad length is also ideal for quick variations, experimental manipulations, and for eye-tracking data collection62. The length of each video is about one minute.</li>
	</ol>
	</li>
	<li>To ensure that emotional and rational appeals are manipulated successfully, have people watch and rate a pool of preselected ads based on these appeals. Here, thirteen coders majoring in advertising were used.
	<ol>
		<li>To maximize manipulation, choose videos with the highest scores in either categories to be the experimental stimuli. The final selected ads represented eight kinds of products. The same product has an emotional ad and a rational ad.</li>
	</ol>
	</li>
	<li>Make the barrage video. There are two ways to make a barrage video for research.
	<ol>
		<li>Upload the video to a barrage website, such as the B station at Bilibili (https://www.bilibili.com/). Have participants log into their accounts to watch, and comment. The uploader controls the position of the barrage, usually at the upper third of the screen.</li>
		<li>Alternatively, use video editing software to convert barrage into subtitles, so comments can be manually added to the video barrage area. The ready-made video can then be called in the data-collection process. This experiment used the second method.</li>
	</ol>
	</li>
	<li>Produce four presentation orders for the experiment to randomize presentational effects. In this design, each participant only saw one version of a particular video, either emotional or rational appeal of an ad, and each saw eight ads altogether.</li>
</ol>
3. Eye tracking protocol
<ol>
	<li>Eye tracking procedure
	<ol>
		<li>Use a commercial eye tracker in the study. Set the default setting for the tracker gaze sample rate at 60 Hz per second.</li>
		<li>Place a 24-inch computer screen 50 cm from the participant&#8217;s chair. Attach the eye tracker to the computer.</li>
		<li>Invite the participant in the lab. Ask the participant to read and sign an informed consent form. Ask the participant to sit comfortably in front of the test computer.</li>
		<li>Have the experimenter check, and adjust the chair height if necessary, to make sure that the TV screen is at the participant&#8217;s eye level.</li>
		<li>Ask participants to sit still to complete a calibration task to ensure that data collected during the experiment are accurate. Inform participant that a 5-point calibration is necessary to achieve the highest accuracy in data collection, and to track participants&#8217; gaze within 2&#176; of accuracy.</li>
		<li>Ask participant to follow a moving red dot on the computer screen with both eyes and to fixate on it when it stopped. If a participant looks away during calibration, then repeat the process.</li>
		<li>Check the tracker software to see if a participant misses a calibration point. If so, repeat the calibration.</li>
		<li>Ask the participant to click the left mouse button to start an exercise test to familiarize them with the experimental procedure.</li>
		<li>Start the main experiment and tell the participant that he/she will see a red &#34;+&#34; sign in the middle of the screen, which lasts 500 ms, alerting to the start of the experiment.</li>
		<li>Ask participant to watch the first video while eye tracking is on.</li>
		<li>After the first video, ask participant to complete a questionnaire page, which automatically pops up. Have participants complete a battery of evaluative measures on satisfaction with the viewing by clicking the left mouse button and choosing ratings.</li>
		<li>Ask participant to take a break if desired or continue to another video (see Figure 3, credited to computers in Human Behaviors (CHB) for a flowchart of the experiment).</li>
		<li>Repeat the procedure seven more times for each participant to complete the entire protocol and the eight ads for each.</li>
		<li>Thank, debrief, and pay participant CN&#165;10.</li>
	</ol>
	</li>
</ol>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/61089/61089fig03.jpg" /> 
Figure 3: Experimental flowchart <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/61089/61089fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<ol start="2">
	<li>Eye-tracking data and self-reported data analysis 
	NOTE: The eye tracker recorded the whole length of the experiment, including the segments of ad watching and questionnaire responding. In this recording, eye movement data were superimposed over video. Two measures were used in this study, including fixations and durations. A fixation is where the eyes were relatively still, with the central foveal vision being held in place so the human visual system could process the information in that point. A fixation in an eye-tracker was typically defined as a succession of raw gaze points where the velocity was below a pre-defined threshold in the tracker&#8217;s gaze filter. More than 60 ms of gaze would be considered as a fixation. Duration, on the other hand, was the length of time between the onset of first gaze point and the last gaze point that make up a fixation.
	<ol>
		<li>Slice the entire recording into eight segments corresponding to each ad watching segment. Each clip still contains the original ad and eye movement data.</li>
		<li>On the sliced video, use the tracker software to draw an area of interest (AOI) to distinguish between eye movement data in the barrage area and non-barrage area. The top third was the AOI for barrage, and the lower two-thirds were the AOI for non-barrage.</li>
		<li>First count the number of fixations for each video segment, and separate them into fixations on the barrage AOI and non-barrage AOI.</li>
		<li>Calculate durations.</li>
		<li>Compare duration and the number of fixations at the barrage AOI relative to the whole scene. These two measures allow researchers to infer where participants are focusing and what elements they are paying attention to while watching the videos.</li>
		<li>Analyze self-reported data to examine participants&#39; satisfaction towards video</li>
	</ol>
	</li>
</ol>

The authors declare that they have nothing to disclose.

In this study, eye tracking technology and self-reports are combined to test the validity of the proposed AMSM model. Previous studies mainly used self-reports to explore the relationship between loneliness and media use at the mercy of participants&#8217; ability to articulate. These offline methods fail to understand the psychological processes while watching barrage videos. In this study, eye tracking technology was used to record duration and fixation so researchers could reliably infer how much cognitive resources the participants invested in different video content. The technology is noninvasive, so participants watched the videos without interruption. Results supported the proposed AMSM.
Specifically, this study provided evidence that lonely people gravitated toward emotional content. When emotional content was watched and being discussed (i.e., with barrage), it was more attention getting as measured in gaze duration and fixation counts. In contract, data did not show such effects on people who were not lonely. Equally important, satisfaction evaluation after exposure showed that lonely people were more satisfied with emotional, barrage video, whereas no such conclusion could be drawn for people low in loneliness.
Theoretically, data from this study did not support traditional models of media consumption, claiming either that people used media for the sake of compensating emotional needs, according to the deficiency model, or viewers consumed for broader, more general purposes such as learning and diversion, according to the general use model. Instead, data supported the proposed AMSM model, which posited that emotional content activated viewers&#8217; affective needs, encouraging them to seek out interactive features such as barrage in media to fulfill such needs.
Methodologically, even though eye tracking has been used in communication in a variety of areas such as news consumption and advertising effectiveness, most of these studies explore physical attributes of media such as story placement, layout designs, and product features to explore attention-getting attributes. This study used eye-tracking creatively in the area of affect, by measuring personality variables related to loneliness, and manipulating media variables of emotional content and interactive barrage. It is the researchers&#8217; belief that eye tracking did not have to focus on the obvious, it can also be used to investigate psychological processes. The new barrage format also offers a new and interesting arena for research.
However, this study also has some limitations. For the sake of manipulation convenience, ads were used instead of the usual entertainment content for barrage. Normally, barrage viewers were highly involved in story plots to engage in barrage viewing and participation. Viewing a number of ads seemed artificial. Also, because barrage was pre-produced before the experiment, participants of this study did not have a chance to really engage in barrage activities. As such, the activation process as proposed in AMSA was not specifically investigated. Future research may want to focus on the before and after emotional states to infer whether activation occurs. Also in this study, barrage appeared in the top part of the screen and its crawling direction was predetermined to be from left to right. In real life, barrage texts appear in different styles. Future research may need to allow natural barrage activity to increase external validity.
In sum, the combination of this approach&#8212;objective eye tracking data and subjective self-reporting data&#8212;allows the researchers to not only untangle the viewing mechanism of lonely on barrage videos, but also to identify the underlying mechanisms of attention allocation. Our hope is that it can serves as a starting point for more such studies.

Eye-tracking technique 
The eye is often referred to as a window of the mind1,2. Eighty percent of human information intake is obtained visually3. Since the 19th century, people began to study human psychological activities by directly observing eye movements of participants. Miles invented the peephole to make observations when participants read4. In the study, the experimenter sat opposite a participant and observed the participant&#8217;s eye movements through a small hole in the middle of the reading material. Since then, technology has vastly improved. Currently, state-of-the-art eye-movement tracking devices mainly focus on electric current-recording, magnetic-induction and optical-recording, which includes corneal reflection and iris-scleral reflection methods5,6,7. The noninvasive features widely prevalent today make eye movement recording more natural, enhancing ecological validity. Today, eye movement techniques generally refer to the use of computer-controlled eye tracking to record and analyze the positioning of the participant&#39;s eye and the forms of eye movements during viewing of visual material.
Many theoretical perspectives regarding eye movements have matured over the years. These include the vision buffer processing model, the parafoveally processing model, the E-Z Reader model, the immediately processing model and the eye-mind processing model8,9,10. The immediately processing model holds that the processing of viewing content at all levels is not delayed, but occurs in real-time. The eye-mind processing model focuses on text information and holds that as long as one is processing a word, one would look at it. Put in another way, the word that one processes is exactly the very word one is looking at. The processing time of a word is the total fixation time of the participant&#8217;s eye.
There are three basic types of human eye movements: fixation, saccades and pursuit movements8,11,12. The fixation duration and fixation count usually reflect the extent to which the participant exerts cognitive resources to the content being viewed. Saccade refers to the movement from one gaze point to another. Retrospective saccade can be used as an indicator of the processing sophistication in the encoding process. Regression saccade indicates a deeper processing of an area after the first gaze of the key areas, reflecting the difficulty or interest of the content in that area. Pursuit movements are usually performed when there is visual noise, and the eye seeks out a point of interest.
On the other hand, we can also measure pupil size and blink frequency; both reflect people&#8217;s psychological activities13,14,15. For example, there is a relationship between pupil size and the specific task difficulty, motivation, interest, attitude, and fatigue. At present, the relationship between pupil size and emotional valence is not clear16. However, if a researcher combines pupil size with other indicators, such as electroencephalogram (EEG), the accuracy would be greatly improved17. For blink indicators, according to the hedonic-blink hypothesis, the decrease of blink frequency is usually associated with happy emotional thoughts, while the increase of blink frequency is associated with unhappy emotional states18.
The application of eye-tracking technology is extremely broad, including reading strategies, visual information processing, compulsive behaviors, to even artistic intentions. The application in the field of reading is the most mature. In communication, eye-tracking is useful in news consumption studies and advertising effectiveness research. For example, a large number of eye movement experiments have explored both exogenous and endogenous factors in advertising19,20, with the former exploring the physical characteristics of advertisements such as size, pattern, color, position, originality and repeated presentations21,22, and the latter exploring individual factors such as product involvement, product motivation, prior knowledge and brand familiarity23,24,25,26,27.
In addition, eye tracking technology is widely used in many other fields, such as human-computer interaction and usability research28,29,30,31; skills transfer32; infant and child development research33,34,35,36; marketing and online consumer behavior research37,38,39; and packaging design40,41, among others.
In addition to being used alone, eye tracking is often combined with other multimodal measurement techniques. For example, researchers can combine eye movement data with other physiological indicators such as EEG, skin electrical response, heart rate, skin temperature, facial expression, etc. In this way, users&#39; emotional responses to different kinds of information can be studied more effectively42.
Eye movement technology can also be integrated into other technologies. For example, it is being used with augmented reality technology30. At present, the integration of eye movement technology and virtual reality (VR) technology is worthy of attention. On the one hand, such integration can promote the rapid development of VR related equipment. For example, the graphics processing unit (GPU) of most VR equipment is overburdened and consumes a lot of energy. Eye tracking technology can detect the audience&#39;s fixation point in real time. VR devices only need to focus on rendering this area and ignore other areas. This can significantly reduce power consumption and GPU rendering load. On the other hand, the combination can enhance its interaction features, and improve the immersion and involvement of VR users. For example, players can use eye movement instead of hands to complete operations in a game43. In addition, researchers can implement eye movement testing in a simulated environment. For example, in consumer behavior research, researchers do not need to take subjects to a real shopping mall or use real products, but test them in virtual research scenarios, as shown in Figure 1.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/61089/61089fig01.jpg" /> 
Figure 1: Virtual research scenario <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/61089/61089fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
Barrage video 
Barrage is a term originating from the military. In order to intercept moving targets, multiple artillery guns are used to fire and to form a curtain composed of high-density projectiles before the moving targets44. The term &#34;barrage&#34; is used in video viewing to describe a format and a phenomenon in which audience members of a particular video comment and displaying such comments on the video screen while watching a video, thus forming a comment wall curtain, as shown in Figure 2. This format of interacting and commenting is also referred to as overlaid comments45,46. Barrage videos first appeared on the &#34;niconico&#34; video website in Japan. The most popular websites in China are AcFun (Station A, www.acfun.cn) and Bilibili (Station B, www.bilibili.com). Many other streaming video sites in China, such as Youku, Tencent, LeTV and iQiyi, have also added the barrage sending and viewing functions. Barrage videos have attracted many Chinese viewers, and the attention of researchers in many disciplines.
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/61089/61089fig02.jpg" /> 
Figure 2: Comment wall curtain <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/61089/61089fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
Viewers of barrage videos can watch and interact with others by commenting to possibly meet a few psychological needs. The audience of such videos may share similarities in age, background and interests, so they may develop a sense of identity and belonging. There is research, for example, that shows AcFun and Bilibili are home to the &#8220;quadratic element&#8221; genre enthusiasts, with its audience being mainly between 17 and 25 years old47,48. These two websites have set up an extremely strict membership review system in that membership is only granted after acing an entrance exam by answering questions such as &#34;What is the height of the heroine of the Thousands of Defenders in Hurricane Butler?&#34; &#34;How many cruisers have been dispatched by Gonon in the Battle of Rum?&#34; Such criteria basically weed out most of the &#34;heterogeneous&#34; audience who want to &#34;invade&#34; the group.
Related to this study, barrage video creates a sense of crowd watching, which is particularly meaningful for lonely individuals. Because the content of the barrage is highly related to the plot, and comments are immediate responses, it gives the audience an illusion of watching with others even though they may be watching alone physically. This sense of companionship has been shown to alleviate loneliness49.
Barrage videos can also afford entertainment of a different kind in that viewers are not only consuming but also creating video content by blending sometime serious film making with fun plays. Barrage viewers can also find a sanctuary away from reality50,51 in which they can vent their anxiety and engage in emotional catharsis in a safe environment52, or showcase their personalities, demonstrate narcissism by getting others&#8217; attention, and even bypass conformist norms in the real world53.
Viewing mechanism of lonely audience on barrage video 
Barrage video viewing is an ideal platform to study media use and loneliness for its emotional support afforded by the venue. In this study, the researchers find previous conceptualizations of media exposure to be inadequate and therefore offer an activation and match satisfaction model (AMSM) to explain the psychological underpinnings of viewing barrage, especially by loners. In previous research, there are two perspectives that explain media use. The deficiency paradigm holds that loners, for lack of companionship, would devote more cognitive resources to the barrage content during watching, to seek company and to compensate loneliness. The global-use paradigm maintains that media use is prevalent, and that it satisfies generic and interpersonal needs in general. So regardless of emotional state, all viewers would pay attention to barrage, and they obtain different gratifications54. However, AMSM posits that emotional content serves to activate lonely audiences&#39; interpersonal needs, and they would take initiatives to search for interactive and interpersonal communication elements, such as barrage content in the viewing process, and devote more attention to such components. The degree that these elements satisfy their emotional needs determines the degree of satisfaction they obtain.
In order to understand the mechanism of barrage video watching by lonely people, one needs to know the amount of cognitive resources that loners invest in different media content and how they satisfy their needs. However, such data cannot be reliably obtained by traditional participative reporting methods. Cognitive resources allocation operates consciously and subconsciously. It is a tall order that audience members articulate to which parts of content they invest more cognitive resources. To accomplish this, a suitable research methodology is needed to record the viewing process, and to distinguish the amount of attention to different parts of content, in addition to measuring the corresponding satisfaction from the viewing process.
For these reasons, this project tracked participants&#8217; eye movements, as measurements of attention and the degree of cognitive resources allocated. The follow-up Likert scale questions were designed to measure participants&#39; degree of satisfaction from exposure. Eye-tracking is a noninvasive technology that has high temporal and spatial resolutions, allowing recording while participants process continuous visual stimuli without distraction55,56. In this study, duration time and fixation counts are used as measures of attention. Duration time refers to the length of attention, and fixation count refers to the number of gazes at a particular area of the video material. Both of these eye movement indicators have been shown to be valid measures of processing thoroughness, reflecting the cognitive resources allocated by individuals57,58. Results of gaze probability, for example, allow researchers to infer attributes in the video that were important to the participant. For self-reported measures, the researchers use a 7-point Likert scale, asked and answered immediately after watching.
Based on arguments from previous explicated perspectives, the researchers designed a 2 (audience type) x 2 (ad appeal) x 2 (barrage) mixed experimental study and hypothesized the amount of attention that normal and lonely participants would pay to barrage video. Audience type (lonely and normal) was a between factor. Ad appeal had two levels, with either emotional ads or rational ads. Barrage also had two levels, denoting video that either had barrage or not. The last two were within subject factors. The general hypotheses were that lonely audience would pay more attention to emotional ads than to rational ads, and they would pay more attention to barrage than to non-barrage, whereas for normal audience, there was no such difference. Satisfaction ratings followed the same patterns. The Chen et al.59 original paper detailed all these hypotheses.

<table><tbody><tr><td>eyetracker</td><td>Tobii</td><td>T60</td><td></td></tr><tr><td>questionnaire</td><td></td><td></td><td></td></tr></tbody></table>

loneliness assuaged: eye-tracking an audience watching barrage videos

Researchers usually theorize media exposure based on assumptions of legacy media. However, a new interactive video viewing format, in this case barrage video where viewers&#8217; comments are overlaid over visual content, challenges past perspectives. This study proposes an activation and match satisfaction model to study viewing behaviors of lonely people and to challenge previous claims. It presents a protocol to examine the mechanism of how loners use barrage videos by combining eye tracking and self-report measures. Eye tracking documents the audience&#39;s conscious and subconscious watching behaviors in real time and allows for inference of the amount of allocated cognitive resources in response to rational and emotional content. The self-report gauges the amount of satisfaction obtained. Overall, results from the measures supported an activation and match satisfaction model regarding loners and their barrage video viewing behaviors. Implications are discussed.

Repeated measures MANOVAs were conducted using duration and fixation as dependent variables, which indicated attention. Results confirmed proposed hypotheses that lonely participants&#8217; gaze stayed on barrage longer than on non-barrage areas when the emotional ads were present. However, when the rational ads were viewed, there was no difference. Such data pattern did not replicate for the low loneliness participants, whose attention remained statistically non-significant when watching emotional ads and rational ads. That is, there was no difference between the gaze probability to the barrage area of interest and the gaze probability of the non-barrage area of interest. The patterns of eye movement data were only in line with the expectations of AMSM. Both duration and fixation count produced significant statistical results59. See Figure 4, credited to CHB, below for the representative fixation count results.
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/61089/61089fig04.jpg" /> 
Figure 4: Percentage of fixation count in the barrage <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/61089/61089fig04large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
For the significant interaction of loneliness, ad appeal, and barrage, pairwise comparisons were computed. The results showed that lonely subjects gazed at the emotional/barrage area the most, whereas a normal audience exhibited no gaze differences within either emotional or barrage condition.
Self-reported data and analyses 
The participants&#8217; satisfaction level was measured by using established satisfaction items from previous research63,64. Satisfaction with watched videos largely replicated the duration and fixation results, and as proposed by AMSM, and as shown in Table 1, credited to CHB59. Results showed that a lonely audience was more satisfied watching emotional ads than rational ads, and with barrage than non-barrage, whereas for a normal audience, there were no statistical differences for both interactions.
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td colspan="5">Evaluation results of advertising videos (M &#177; SD)</td>
		</tr>
		<tr>
			<td></td>
			<td colspan="2">Story-base ads</td>
			<td colspan="2">Hard-selling ads</td>
		</tr>
		<tr>
			<td>Audience type</td>
			<td>Barrage video</td>
			<td>Non-barrage video</td>
			<td>Barrage video</td>
			<td>Non-barrage video</td>
		</tr>
		<tr>
			<td>Loneliness</td>
			<td>4.84&#177;0.69</td>
			<td>4.45&#177;0.73</td>
			<td>4.30&#177;0.87</td>
			<td>4.30&#177;1.06</td>
		</tr>
		<tr>
			<td>Normal</td>
			<td>4.75&#177;0.67</td>
			<td>4.62&#177;0.80</td>
			<td>4.14&#177;0.79</td>
			<td>4.21&#177;0.87</td>
		</tr>
		<tr>
			<td colspan="5">Note: revised from Chen &#38; Zhou (2019) published in CHB</td>
		</tr>
	</tbody>
</table>
Table 1: Evaluation results of advertising videos
Video 1: Sample barrage ad <a href="https://cloudflare-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/61089/ad_sample.wmv" target="_blank">Please click here to download this video.</a>

Watch this Scientific Journal Video about Loneliness Assuaged: Eye-Tracking an Audience Watching Barrage Videos at JoVE.com

Loneliness Assuaged: Eye-Tracking an Audience Watching Barrage Videos

The study proposes an activation-match model to study how loneliness is mitigated when a lonely audience watches barrage videos of rational and emotional appeals. The protocol uses eye tracking to document duration and fixation, accounting for the degree of satisfaction when emotional needs are appeased by content and barrage.

Researchers usually theorize media exposure based on assumptions of legacy media. However, a new interactive video viewing format, in this case barrage ...

loneliness-assuaged-eye-tracking-an-audience-watching-barrage-videos

Nanyang Technological University

Research

JoVE Journal

Behavior

4.1K Views.  Jinan University. The study proposes an activation-match model to study how loneliness is mitigated when a lonely audience watches barrage videos of rational and emotional appeals. The protocol uses eye tracking to document duration and fixation, accounting for the degree of satisfaction when emotional needs are appeased by content and barrage.

Video: Loneliness Assuaged: Eye-Tracking an Audience Watching Barrage Videos

Automated Interactive Video Playback for Studies of Animal Communication

Video playback is a widely-used technique for the controlled manipulation and presentation of visual signals in animal communication. In particular, parameter-based computer animation offers the opportunity to independently manipulate any number of behavioral, morphological, or spectral characteristics in the context of realistic, moving images of animals on screen. A major limitation of conventional playback, however, is that the visual stimulus lacks the ability to interact with the live animal. Borrowing from video-game technology, we have created an automated, interactive system for video playback that controls animations in response to real-time signals from a video tracking system. We demonstrated this method by conducting mate-choice trials on female swordtail fish, Xiphophorus birchmanni. Females were given a simultaneous choice between a courting male conspecific and a courting male heterospecific (X. malinche) on opposite sides of an aquarium. The virtual male stimulus was programmed to track the horizontal position of the female, as courting males do in the wild. Mate-choice trials on wild-caught X. birchmanni females were used to validate the prototype's ability to effectively generate a realistic visual stimulus.

Video playback is a widely used technique in animal behavior. We created and evaluated a program that applies rules-based, interactive playback of 3-D computer animations in response to real-time, automated data on subject behavior.

Video playback is a widely-used technique for the controlled manipulation and presentation of visual signals in animal communication. In particular, ...

Neuroscience

Brain Imaging Investigation of the Neural Correlates of Observing Virtual Social Interactions

The ability to gauge social interactions is crucial in the assessment of others&#x2019; intentions. Factors such as facial expressions and body language affect our decisions in personal and professional life alike 1. These "friend or foe" judgements are often based on first impressions, which in turn may affect our decisions to "approach or avoid". Previous studies investigating the neural correlates of social cognition tended to use static facial stimuli 2. Here, we illustrate an experimental design in which whole-body animated characters were used in conjunction with functional magnetic resonance imaging (fMRI) recordings. Fifteen participants were presented with short movie-clips of guest-host interactions in a business setting, while fMRI data were recorded; at the end of each movie, participants also provided ratings of the host behaviour. This design mimics more closely real-life situations, and hence may contribute to better understanding of the neural mechanisms of social interactions in healthy behaviour, and to gaining insight into possible causes of deficits in social behaviour in such clinical conditions as social anxiety and autism 3.

This article demonstrates an experimental design in which whole-body animated characters are used in conjunction with functional magnetic resonance imaging (fMRI) to investigate the neural correlates of observing virtual social interactions.

The ability to gauge social interactions is crucial in the assessment of others&#x2019; intentions. Factors such as facial expressions and body language ...

Methods to Test Visual Attention Online

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative data samples than can typically be collected on college campuses. However, before such methods can be widely adopted, a number of technological challenges must be overcome &#8211; in particular in experiments where tight control over stimulus properties is necessary. Here we present methods for collecting performance data on two tests of visual attention. Both tests require control over the visual angle of the stimuli (which in turn requires knowledge of the viewing distance, monitor size, screen resolution, etc.) and the timing of the stimuli (as the tests involve either briefly flashed stimuli or stimuli that move at specific rates). Data collected on these tests from over 1,700 online participants were consistent with data collected in laboratory-based versions of the exact same tests. These results suggest that with proper care, timing/stimulus size dependent tasks can be deployed in web-based settings.

To replicate laboratory settings, online data collection methods for visual tasks require tight control over stimulus presentation. We outline methods for the use of a web application to collect performance data on two tests of visual attention.

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative ...

Eye-Tracking Control to Assess Cognitive Functions in Patients with Amyotrophic Lateral Sclerosis

Amyotrophic lateral sclerosis (ALS) is a neurodegenerative disorder with pathological involvement of upper and lower motoneurons, subsequently leading to progressive loss of motor and speech abilities. In addition, cognitive functions are impaired in a subset of patients. To evaluate these potential deficits in severely physically impaired ALS patients, eye-tracking is a promising means to conduct cognitive tests. The present article focuses on how eye movements, an indirect means of communication for physically disabled patients, can be utilized to allow for detailed neuropsychological assessment. The requirements, in terms of oculomotor parameters that have to be met for sufficient eye-tracking in ALS patients are presented. The properties of stimuli, including type of neuropsychological tests and style of presentation, best suited to successfully assess cognitive functioning, are also described. Furthermore, recommendations regarding procedural requirements are provided. Overall, this methodology provides a reliable, easy to administer and fast approach for assessing cognitive deficits in patients who are unable to speak or write such as patients with severe ALS. The only confounding factor might be deficits in voluntary eye movement control in a subset of ALS patients.

Cognitive deficits are common in about one third of patients with amyotrophic lateral sclerosis, a neurological condition leading to progressive impairments in speech and movement abilities. To conduct cognitive tests in patients unable to speak or write a reliable and easy to administer eye-tracking paradigm was developed.

Amyotrophic lateral sclerosis (ALS) is a neurodegenerative disorder with pathological involvement of upper and lower motoneurons, subsequently leading to ...

Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment

As individuals increasingly live in cities, methods to study their everyday movements and the data that can be collected becomes important and valuable. Eye-tracking informatics are known to connect to a range of feelings, health conditions, mental states and actions. But because vision is the result of constant eye-movements, teasing out what is important from what is noise is complex and data intensive. Furthermore, a significant challenge is controlling for what people look at compared to what is presented to them.
The following presents a methodology for combining and analyzing eye-tracking on a video of a natural and complex scene with a machine learning technique for analyzing the content of the video. In the protocol we focus on analyzing data from filmed videos, how a video can be best used to record participants&#39; eye-tracking data, and importantly how the content of the video can be analyzed and combined with the eye-tracking data. We present a brief summary of the results and a discussion of the potential of the method for further studies in complex environments.

The objective of the protocol is to detail how to collect video data for use in the laboratory; how to record eye-tracking data of participants looking at the data and how to efficiently analyze the content of the videos that they were looking at using a machine learning technique.

As individuals increasingly live in cities, methods to study their everyday movements and the data that can be collected becomes important and valuable. ...

Environment

Using Electroencephalography Measurements and High-quality Video Recording for Analyzing Visual Perception of Media Content

This article explores a method to detect differences in visual perception in humans. The method used is based on the psychological (or "cognitive") function of eyeblinks. Participants' eyeblinks are detected and acquired while watching videos specifically created for the investigation. The detection and acquisition of eyeblinks are carried out with the help of a 20-channel electroencephalographic (EEG) wireless device. The international 10-20 system for electrode placement is followed. A high-definition (HD) video camera is used to record participants' facial expressions, for contrast purposes. Instead of using pre-existing media content, purpose-made video content has been created following specific criteria of interest for this investigation, with stimuli enabling researchers to manage the precise parameters of interest. Otherwise, results could be contaminated with uncontrolled variables. The synchronization of the presentation of video stimuli with EEG recordings needs to be done in milliseconds. Analysis of collected data is performed with robust software for working with big matrices. Statistically significant differences in eyeblink rate related to media professionalization and editing style are found with the reported experimental procedures.

We present detection, acquisition, and analysis of eyeblink rates while watching media content.

This article explores a method to detect differences in visual perception in humans. The method used is based on the psychological (or "cognitive") ...

Comparing Eye-tracking Data of Children with High-functioning ASD, Comorbid ADHD, and of a Control Watching Social Videos

Children with autism spectrum disorders (ASD) are known to have sensory-perceptual processing deficits that weaken their abilities to attend and perceive social stimuli in daily living contexts. Since daily social episodes consist of subtle dynamic changes in social information, any failure to attend to or process subtle human nonverbal cues, such as facial expression, postures, and gestures, might lead to inappropriate social interaction. Traditional behavioral rating scales or assessment tools based on static social scenes have limitations in capturing the moment-to-moment changes in social scenarios. An eye-tracking assessment, which can be administered in a video-based mode, is therefore preferred, to augment clinical observation. In this study, using the single-case comparison design, the eye-tracking data of three participants, a child with autism spectrum disorder (ASD), another with comorbid attention deficit-hyperactive disorder (ADHD), and a neurotypical control, are captured while they view a video of social scenarios. The eye-tracking experiment has helped answer the research question: How does social attention differ between the three participants? By predefining areas of interest (AOIs), their visual attention on relevant or irrelevant social stimuli, how fast each participant attends to the first social stimuli appearing in the videos, for how long each participant continues to attend to those stimuli within the AOIs, and the gaze shifts between multiple social stimuli appearing concurrently in the same social scene are captured, compared, and analyzed in a video-based eye-tracking experiment.

This is a qualitative comparative case study analysis of eye-tracking data on the first moments of social video scenes as viewed by three participants: one with autism spectrum disorder, one with comorbid attention deficit-hyperactive disorder, and one neurotypical control.

Children with autism spectrum disorders (ASD) are known to have sensory-perceptual processing deficits that weaken their abilities to attend and perceive ...

A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency sensors, such as eye-trackers, electrodermal activity wristbands, EEG bands, and motion sensors provide observations at the millisecond level. This level of precision allows researchers to collect large datasets on social interactions. In this paper, I discuss how multiple eye-trackers can capture a fundamental construct in social interactions, joint visual attention (JVA). JVA has been studied by developmental psychologists to understand how children acquire language, learning scientists to understand how small groups of learners work together, and social scientists to understand interactions in small teams. This paper describes a methodology for capturing JVA in colocated settings using mobile eye-trackers. It presents some empirical results and discusses implications of capturing microobservations to understand social interactions.

Using multimodal sensors is a promising way to understand the role of social interactions in educational settings. This paper describes a methodology for capturing joint visual attention from colocated dyads using mobile eye-trackers.

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency ...

Measuring Engagement of Spectators of Social Digital Games

The goal of this methodology is to assess explicit and implicit measures of engagement of spectators during social digital games in a group of participants with motion tracking systems. In the context of games that are not confined within a screen, measuring the different dimensions of engagement such as physiological arousal can be challenging. The focus of the study is made on the spectators of the game and the differences in their engagement according to interactivity. Engagement is measured with physiological and self-reported arousal, as well as an engagement questionnaire at the end of the experiment. Physiological arousal is measured with electrodermal activity (EDA) sensors that record the data on a portable device (EDA box). Portability was essential because of the nature of the game, which is akin to a life-size pong and includes many participants that move. To have an overview of the events of the game, three cameras are used to film three angles of the playing field. To synchronize the EDA data with events happening in the game, boxes with digital numbers are used and put in the frames of cameras. Signals are sent from a sync box simultaneously to the EDA boxes and to light boxes. The light boxes show the synchronization numbers to the cameras, and the same numbers are also logged on the EDA data file. That way, it is possible to record EDA of many people that move freely in a large space and synchronize this data with events in the game. In our particular study, we were able to assess the differences in arousal for the different conditions of interactivity. One of the limitations of this method is that the signals cannot be sent farther than 20 meters away. This method is, therefore, appropriate for recording physiological data in games with an unlimited number of players but is restricted to a limited space.

We propose a methodology that enables measuring engagement of spectators in a social digital game combining physiological and self-reported data. As this digital game involves a group of freely moving people, the experience is filmed using a synchronizing technique that links physiological data with events in the game.

The goal of this methodology is to assess explicit and implicit measures of engagement of spectators during social digital games in a group of ...

Eye-tracking Technology and Data-mining Techniques used for a Behavioral Analysis of Adults engaged in Learning Processes

Behavioral analysis of adults engaged in learning tasks is a major challenge in the field of adult education. Nowadays, in a world of continuous technological changes and scientific advances, there is a need for life-long learning and education within both formal and non-formal educational environments. In response to this challenge, the use of eye-tracking technology and data-mining techniques, respectively, for supervised (mainly prediction) and unsupervised (specifically cluster analysis) learning, provide methods for the detection of forms of learning among users and/or the classification of their learning styles. In this study, a protocol is proposed for the study of learning styles among adults with and without previous knowledge at different ages (18 to 69-year-old) and at different points throughout the learning process (start and end). Statistical analysis-of-variance techniques mean that differences may be detected between the participants by type of learner and previous knowledge of the task. Likewise, the use of&#160;unsupervised learning clustering techniques throws light on similar forms of learning among the participants across different groups. All these data will facilitate personalized proposals from the teacher for the presentation of each task at different points in the chain of information processing. It will likewise be easier for the teacher to adapt teaching materials to the learning needs of each student or group of students with similar characteristics.

We present a protocol for a behavioral analysis of adults (ages 18 to 70-year-old) engaged in learning processes, undertaking tasks designed for Self-Regulated Learning (SRL). The participants, university teachers and students, and adults from the University of Experience, were monitored with eye-tracking devices and the data were analyzed with data-mining techniques.