Name: using eye movements recorded in the visual world paradigm to explore the online processing of spoken language
Uploaded: 2018-10-13T12:30:00.000+00:00
Duration: 9 min 27 s
Description: Watch this Scientific Journal Video about Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language at JoVE.com

This research was supported by Science Foundation of Beijing Language and Cultural University under the Fundamental Research Funds for the Central Universities (Approval number 15YJ050003).

The author declares that he has no competing financial interests.

To conduct a visual world study, there are several critical steps to follow. First, researchers intend to deduce the interpretation of the auditorily presented language via participants&#39; eye movements in the visual world. Henceforth, in designing the layout of the visual stimuli, the properties of eye movements in a natural task that potentially affect participants&#39; eye movements should be controlled. The effect of the spoken language on participants&#39; eye movements can then be recognized. Second, acoustic cues in the spoken language are transient and there are no acoustic signatures solely corresponding to certain linguistic category. To correctly time-lock participants&#39; eye-movements with the onset of some linguistic marker, researchers should find an objective and consistent way to define the boundary of some linguistic structure. Third, to correctly map participants&#39; eye rotation around the head with their gaze of regard in the visual world, researchers need to perform one or several runs of calibration, validation, and drift correction processes. Fourth, the data obtained from a visual world study have some peculiar properties, such as being lower and upper bounded, and having autocorrelated errors etc. These peculiar properties should be considered when a method is selected to statistically analyze the data.
A visual world study consists of three essential components: visual display, spoken language, experimental task. Modifications can be made on any component to fulfill researchers&#39; specific purposes. First, a visual display is normally a screening display depicting an array of pictures. But it can also be a screening display depicting an array of printed words32, a schematic scene33,34, or a real world scene containing real objects1,35. Second, the spoken utterances can be a word36, a simple sentence33,34, a semantically complex statement17,37,38, or a dialogue39. Third, in terms of the experimental task, participants are either simply being asked to look at the visual world and listen carefully to the auditory utterances33,34; or are being required to make some behavioral responses, such as acting out the movements described by the auditory utterance1, determining whether or not the auditory utterance applies to the visual display38, or choosing the correct image in the visual display the spoken utterance is talking about17.
The visual world paradigm, compared with other psycholinguistic techniques, has several unique advantages. First, the visual world paradigm can be used in a wide of populations, including those who cannot read and/or who cannot overtly give their behavioral responses, such as preliterate children40,41,42,43,44,45, elderly adults46, and patients (e.g., aphasics)47. This is because the language stimuli are presented in the auditory rather than in the form of written texts; and the language comprehension is inferred from participants&#39; implicit automatic eye movements rather than from their overt behavioral responses. Second, the visual world paradigm is extremely sensitive to fine grained manipulations of the speech signal. This paradigm can be used to study the online processing of most topics in language comprehension at multiple levels, such as the fine grained acoustic phonetic features36,48,49, the properties of words33,34, the linguistic structures1,50, and the logical structures of semantically complex statements like concessives37, biconditionals37, conditionals38, and disjunctions17.
The visual world paradigm, compared with other psycholinguistic technique, also has some potential limitations. First, participants&#39; interpretation of the spoken language is deduced from their eye movements on the visual world, not from the actual interpretation of the language stimuli per se. Henceforth, the language properties that can be studied with the visual world paradigm are limited to the ones that can be visualized, i.e., they should be somehow related to the entities or events in the visual world. Second, the visual world paradigm used is normally more restricted than the actual visual world, with a limited set of pictured referents and a limited set of potential actions. This so-called closed-set problem48&#160;might create task-specific strategies that the observed language processing does not generalize beyond the specific situations created within the experiment. Henceforth, the paradigm might not be sensitive to characteristics of linguistic knowledge and experience lying outside of the closed-set that have been established on a given trial.
The visual world paradigm essentially explores the integration of the information from the visual domain and the information from the auditory domain. Theoretically, any information that can be processed by the two sensational domains can be potentially studied using this paradigm. For example, the visual world can be virtual reality or dynamic videos. The auditory input is not necessarily language, and can be any other format, such as music and&#160;sound from the natural world, etc. Furthermore, this paradigm can be further extended&#160;to explore the integration of information from other domains, rather than the visual domain and the auditory domain. For example, researchers might use this technique to see how participants&#39; fixations in the visual world are affected by different smells, different touches, etc.

Spoken language is a fast, ongoing information flow, which disappears right away. It is a challenge to experimentally study this temporal, rapidly change speech signal. Eye movements recorded in the visual world paradigm can be used to overcome this challenge. In a typical eye tracking study using the visual world paradigm, participants&#39; eye movements to pictures in a display or to real objects in a visual workspace are monitored as they listen to, or produce, spoken language depicting the contents of the visual world1,2,3,4. The basic logic, or the linking hypothesis, behind this paradigm is that comprehending or planning an utterance will (overtly or covertly) shift participants&#39; visual attention to a certain object in the visual world. This attention shift will have a high probability to initiate a saccadic eye movement to bring the attended area into the foveal vision. With this paradigm, researchers intend to determine at what temporal point, with respect to some acoustic landmark in the speech signal, a shift in the participant&#39;s visual attention occurs, as measured by a saccadic eye movement to an object or a picture in the visual world. When and where saccadic eye movements are launched in relation to the speech signal are then used to deduce the online language processing. The visual world paradigm can be used to study both the spoken language comprehension1,2 and production5,6. This methodological article will focus on comprehension studies. In a comprehension study using the visual world paradigm, participants&#39; eye movements on the visual display are monitored as they listen to the spoken utterances talking about the visual display.
Different eye tracking systems have been designed in history. The simplest, least expensive, and most portable system is just a normal video camera, which records an image of the participant&#39;s eyes. Eye movements are then manually coded through frame-by-frame examination of the video recording. However, the sampling rate of such an eye-tracker is relatively low, and the coding procedure is time consuming. Thus, a contemporary commercial eye tracking system normally uses optical sensors measuring the orientation of the eye in its orbit7,8,9. To understand how a contemporary commercial eye-tracking system works, the following points should be considered. First, to correctly measure the direction of the foveal vision, an infrared illuminator (normally with the wavelength around 780-880 nm) is normally laid along or off the optical axis of the camera, making the image of the pupil distinguishably brighter or darker than the surrounding iris. The image of the pupil and/or of the pupil corneal reflection (normally the first Purkinje image) is then used to calculate the orientation of the eye in its orbit. Second, the gaze location in the visual world is actually contingent not only on the eye orientation with respect to the head but also on the head orientation with respect to the visual world. To accurately infer the gaze of regard from the eye orientation, the light source and the camera of the eye-trackers either are fixed with respect to participants&#39; head (head-mounted eye-trackers) or are fixed with respect to the visual world (table-mounted or remote eye-trackers). Third, the participants&#39; head orientation must either be fixed with respect to the visual world or be computationally compensated if participants&#39; head is free to move. When a remote eye-tracker is used in a head-free-to-move mode, the participants&#39; head position is typically recorded by placing a small sticker on participants&#39; forehead. The head orientation is then computationally subtracted from the eye orientation to retrieve the gaze location in the visual world. Fourth, a calibration and a validation process are then required to map the orientation of the eye to the gaze of regard in the visual world. In the calibration process, participants&#39; fixation samples from known target points are recorded to map the raw eye data to gaze position in the visual world. In the validation process, participants are presented with the same target points as the calibration process. The difference existing between the computed fixation position from the calibrated results and the actual position of the fixated target in the visual world are then used to judge the accuracy of the calibration. To further reconfirm the accuracy of the mapping process, a drift check is normally applied on each trial, where a single fixation target is presented to participants to measure the difference between the computed fixation position&#160;and the actual position of the current target.
The primary data of a visual world study is a stream of gaze locations in the visual world recorded at the sampling rate of the eye-tracker, ranging over the whole or part of the trial duration. The dependent variable used in a visual world study is typically the proportion of samples that participants&#39; fixations are situated at certain spatial region in the visual world across a certain time window. To analyze the data, a time window has firstly to be selected, often referred to as periods of interest. The time window is typically time-locked to the presentation of some linguistic events in the auditory input. Furthermore, the visual world is also needed to split into several regions of interest (ROIs), each of which is associated with one or more objects. One such region contains the object corresponding to the correct comprehension of the spoken language, and thus&#160;is often called the target area. A typical way to visualize the data is a proportion-of-fixation plot, where at each bin in a time window, the proportion of samples with a look to each region of interest are averaged across participants and items.
Using the data obtained from a visual world study, different research questions can be answered: a) On the coarse-grain level, are participants&#39; eye movements in the visual world affected by different auditory linguistic input?&#160;b) If there is an effect, what is the trajectory of the effect over the course of the trial? Is it a linear effect or high-order effect?&#160;and c) If there is an effect, then on the fine-grain level, when is the earliest temporal point where such an effect emerges and how long does this effect last?
To statistically analyze the results, the following points should be considered. First, the response variable, i.e., proportions of fixations, is both below and above bounded (between 0 and 1), which will follow a multinomial distribution rather than a normal distribution. Henceforth, traditional statistical methods based on normal distribution such as t-test, ANOVA, and linear (mixed-effect) models10, cannot be directly utilized until the proportions have been transformed to unbounded variables such as with empirical logit formula11 or have been replaced with unbounded dependent variables such as Euclidean distance12. Statistical techniques that do not require the assumption of normal distribution such generalized linear (mixed-effect) models13 can also be used. Second, to explore the changing trajectory of the observed effect, a variable denoting the time-series has to be added into the model. This time-series variable is originally the eye-tracker&#8217;s sampling points realigned to the onset of the language input. Since the changing trajectory typically is not linear, a high-order polynomial function of the time-series is normally added into the (generalized) linear (mixed-effect) model, i.e., growth curve analyses14. Furthermore, participants&#8217; eye positions in the current sampling point is highly dependent on previous sampling point(s), especially when the recording frequency is high, resulting in the problem of autocorrelation. To reduce the autocorrelation between the adjacent sampling points, original data are often down-sampled or binned. In recent years, the generalized additive mixed effect models (GAMM) have also been used to tackle the autocorrelated errors12,15,16. The width of bins varies among different studies, ranging from several milliseconds to several hundred milliseconds. The narrowest bin a study can choose is restricted by the sampling rate of the eye tracker used in the specific study. For example, if an eye tracker has a sampling rate of 500 Hz, then the width of the time window cannot be smaller than 2 ms = 1000/500. Third, when a statistical analysis is repeatedly applied to each time bin of the periods of interest, the familywise error induced from these multiple comparisons should be tackled. As we described earlier, the trajectory analysis informs the researcher whether the effect observed on the coarse-grain level is linear with respect to the changing of the time, but does not show when the observed effect begins to emerge and how long the observed effect lasts. To determine the temporal position when the observed difference starts to diverge, and to figure out the duration of the temporal period that the observed effect lasts, a statistic analysis has to be repeatedly applied to each time bin. These multiple comparisons will introduce the so-called familywise error, no matter what statistical method is used. The familywise error is traditionally corrected with Bonferroni adjustment17. Recently, a method called nonparametric permutation test originally used in neuroimaging field18 has been applied to the visual word paradigm19 to control for the familywise error.
Researchers using the visual world paradigm intend to infer the comprehension of some spoken language from participants&#8217; eye movements in the visual world. To ensure the validity of this deduction, other factors that possibly influence the eye movements should be either ruled out or controlled. The following two factors are among the common ones that need to be considered. The first factor involves some systematic patterns in participants&#8217; explanatory fixations independent of the language input, such as the tendency to fixate on the top left quadrat of the visual world, and moving eyes in the horizontal direction being easier than in the vertical direction, etc.12,20 To make sure that the observed fixation patterns are related to the objects, not to the spatial locations where the objects are situated, the spatial positions of an object should be counterbalanced across different trials or across different participants. The second factor that might affect participants&#8217; eye movements is the basic image features of the objects in the visual world, such as luminance contrast, color and edge orientation, among others21. To diagnose this potential confounding, the visual display is normally presented prior to the onset of the spoken language or prior to the onset of the critical acoustic marker of the spoken language, for about 1000 ms. During the temporal period from the onset of the test image to the onset of the test audio, the language input or the disambiguation point of the language input has not been heard yet. Any difference observed between different conditions should be deduced to other confounding factors such as the visual display per se, rather than the language input. Henceforth, eye movements observed in this preview period provide a baseline for determining the effect of the linguistic input. This preview period also allows participants to get familiarized with the visual display, and to reduce the systematic bias of the explanatory fixations when the spoken language is presented.
To illustrate how a typical eye tracking study using the visual world paradigm is conducted, the following protocol describes an experiment adapted from L. Zhan&#160;17&#160;to explore the online processing of semantically complex statements, i.e., disjunctive statements (S1 or S2), conjunctive statements (S1 and S2), and but-statements (S1 but not-S2). In ordinary conservation, the information expressed by some utterances is actually stronger than its literal meaning. Disjunctive statements like Xiaoming&#39;s box contains a cow or a rooster are such utterances. Logically, the disjunctive statement is true as long as the two disjuncts Xiaoming&#39;s box contains a cow and Xiaoming&#39;s box contains a rooster are not both false. Therefore, the disjunctive statement is true when the two disjuncts are both true, where the corresponding conjunctive statement Xiaoming&#39;s box contains a cow and a rooster is also true. In ordinary conversation, however, hearing the disjunctive statement often suggests that the corresponding conjunctive statement is false (scalar implicature); and suggests that the truth values of the two disjuncts are unknown by the speaker (ignorance inference). Accounts in the literature differ in whether two inferences are grammatical or pragmatic processes22,23,24,25,26. The experiment shows how the visual world paradigm can be used to adjudicate between these accounts, by exploring the online processing of three complex statements.

<table><tbody><tr><td>Pixelmator</td><td>Pixelmator Team</td><td>http://www.pixelmator.com/pro/</td><td>image editing app</td></tr><tr><td>Praat</td><td>Open Sourse</td><td>http://www.fon.hum.uva.nl/praat/</td><td>Sound analyses and editting software</td></tr><tr><td>Eyelink 1000plus</td><td>SR-Research, Inc</td><td>https://www.sr-research.com/products/eyelink-1000-plus/</td><td>remote infrared eye tracker&amp;nbsp;</td></tr><tr><td>Experimental Builder</td><td>SR-Research, Inc</td><td>https://www.sr-research.com/experiment-builder/</td><td>eye tracker software&amp;nbsp;</td></tr><tr><td>Data Viewer</td><td>SR-Research, Inc</td><td>https://www.sr-research.com/data-viewer/</td><td>eye tracker software&amp;nbsp;</td></tr><tr><td>R</td><td>Open Sourse</td><td>https://www.r-project.org</td><td>free software environment for statistical computing and graphics</td></tr></tbody></table>

using eye movements recorded in the visual world paradigm to explore the online processing of spoken language

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded via an eye tracker as the participant produces or comprehends a spoken language describing the concurrent visual world. This paradigm has high versatility, as it can be used in a wide range of populations, including those who cannot read and/or who cannot overtly give their behavioral responses, such as preliterate children, elderly adults, and patients. More importantly, the paradigm is extremely sensitive to fine grained manipulations of the speech signal, and it can be used to study the online processing of most topics in language comprehension at multiple levels, such as the fine grained acoustic phonetic features, the properties of words, and the linguistic structures. The protocol described in this article illustrates how a typical visual world eye tracking study is conducted, with an example showing how the online processing of some semantically complex statements can be explored with the visual world paradigm.

Participants&#39; behavioral responses are summarized in Figure 4. As we described earlier, the correct response to a conjunctive statement (S1 and S2) is the big open box, such as Box A in Figure 1. The correct response to a but-statement (S1 but not S2) is the small open box containing the first mentioned animal, such as Box D in Figure 1. Critically, which box is chosen to the disjunctive statement&#160;(S1 or S2) depends on how a disjunctive statement is processed: The small closed box, such as box B in Figure 1, is chosen only when the scalar implicature and the ignorance inferences relating to the&#160; disjunctive statement&#160;are both computed. Figure 4 illustrates that when comprehending a disjunctive statement (S1 or S2), participants compute both the two inferences.
Furthermore, participants&#39; eye fixations on the small closed box, especially when these fixations are followed by the behavioral responses on that box, also suggest that the scalar implicature and the ignorance inferences are computed. The two inferences should have been processed no later than&#160;the earliest temporal point when participants shift their visual attention and fixations on the small closed box. Participants&#39; eye-movements are summarized in Figure 5. As we can see in the panel B, participants&#39; eye-fixations on the small-closed box (box B) don&#39;t increase unless the sentential connective is the disjunctive connective, or. Furthermore, this increase begins no later than the offset of the disjunctive connective. This suggests that both the scalar implicature and the ignorance inferences are computed before the offset of the sentential connective, i.e., immediately upon encountering the disjunctive connective.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/58086/58086fig1.jpg" /> 
Figure 1. An example of the test images used in the experiment. The gray area is the test image&#160;actually being presented to&#160;participants. The blues boxes, the dotted lines, and the pixels denoting the width of the elements are only for the purposes of illustration and are invisible to&#160;participants. (Adapted from experiment one of L. Zhan 17 with permission). <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/58086/58086fig1large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/58086/58086fig2.jpg" /> 
Figure 2. An example of the test sentences used in the experiment (Reprinted from L. Zhan17 with permission). <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/58086/58086fig2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/58086/58086fig3.jpg" /> 
Figure 3. The hierarchical structure of a typical eye-tracking experiment using the visual world paradigm. The audio illustrated in the image are the English translations of the Mandarin Chinese used in the experiment. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/58086/58086fig3large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/58086/58086fig4.jpg" /> 
Figure 4. Participants&#39; behavioral responses recorded in the experiment (Adapted from experiment one of L. Zhan17 with permission). <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/58086/58086fig4large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 5" class="xfigimg" src="/files/ftp_upload/58086/58086fig5.jpg" /> 
Figure 5. Participants&#39; eye movements observed in the experiment. The onset and offset of the sentential connectives are signified by two dashed vertical lines. A significant difference existing between the disjunctive and the baseline condition at certain sampling point is signified by the gray area (p &#60; .05, Bonferroni adjusted) (Adapted from L. Zhan17 with permission).&#160;<a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/58086/58086fig5large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

Watch this Scientific Journal Video about Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language at JoVE.com

Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

The visual world paradigm monitors participants' eye movements in the visual workspace as they are listening to or speaking a spoken language. This paradigm can be used to investigate the online processing of a wide range of psycholinguistic questions, including semantically complex statements, such as disjunctive statements.

In a typical eye tracking study using the visual world paradigm, participants' eye movements to objects or pictures in the visual workspace are recorded ...

using-eye-movements-recorded-visual-world-paradigm-to-explore-online

Nanyang Technological University

Research

JoVE Journal

Behavior

9.9K Views.  Beijing Language and Culture University. Deze methode kan helpen bij het beantwoorden van belangrijke vragen in online verwerking van gesproken taal, met inbegrip van semantisch complexe zinnen, zoals disjuncties en conditionalen. Het belangrijkste voordeel van deze techniek is dat het kan worden uitgevoerd op een breed scala van populaties om de meeste onderwerpen te bestuderen op gebieden van taalverwerking. Het aantonen van de procedure zal Xu Jing, een postgraduaat student.Cao Yifan, Wang Meng, Yang Xiaotong, Zhan Min en ...

Video: Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

Using Eye Movements to Evaluate the Cognitive Processes Involved in Text Comprehension

The present article describes how to use eye tracking methodologies to study the cognitive processes involved in text comprehension. Measuring eye movements during reading is one of the most precise methods for measuring moment-by-moment (online) processing demands during text comprehension. Cognitive processing demands are reflected by several aspects of eye movement behavior, such as fixation duration, number of fixations, and number of regressions (returning to prior parts of a text). Important properties of eye tracking equipment that researchers need to consider are described, including how frequently the eye position is measured (sampling rate), accuracy of determining eye position, how much head movement is allowed, and ease of use. Also described are properties of stimuli that influence eye movements that need to be controlled in studies of text comprehension, such as the position, frequency, and length of target words. Procedural recommendations related to preparing the participant, setting up and calibrating the equipment, and running a study are given. Representative results are presented to illustrate how data can be evaluated. Although the methodology is described in terms of reading comprehension, much of the information presented can be applied to any study in which participants read verbal stimuli.

The present article describes how to use eye tracking methodologies to study the cognitive processes involved in text comprehension. Descriptions of eye tracking equipment, how to develop experimental stimuli, and procedural recommendations are included. The information presented can be applied to most any study using verbal stimuli.

The present article describes how to use eye  tracking methodologies to study the cognitive processes involved in text  comprehension. Measuring eye ...

Methods to Test Visual Attention Online

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative data samples than can typically be collected on college campuses. However, before such methods can be widely adopted, a number of technological challenges must be overcome &#8211; in particular in experiments where tight control over stimulus properties is necessary. Here we present methods for collecting performance data on two tests of visual attention. Both tests require control over the visual angle of the stimuli (which in turn requires knowledge of the viewing distance, monitor size, screen resolution, etc.) and the timing of the stimuli (as the tests involve either briefly flashed stimuli or stimuli that move at specific rates). Data collected on these tests from over 1,700 online participants were consistent with data collected in laboratory-based versions of the exact same tests. These results suggest that with proper care, timing/stimulus size dependent tasks can be deployed in web-based settings.

To replicate laboratory settings, online data collection methods for visual tasks require tight control over stimulus presentation. We outline methods for the use of a web application to collect performance data on two tests of visual attention.

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative ...

VisualEyes: A Modular Software System for Oculomotor Experimentation

Eye movement studies have provided a strong foundation forming an understanding of how the brain acquires visual information in both the normal and dysfunctional brain.1 However, development of a platform to stimulate and store eye movements can require substantial programming, time and costs. Many systems do not offer the flexibility to program numerous stimuli for a variety of experimental needs. However, the VisualEyes System has a flexible architecture, allowing the operator to choose any background and foreground stimulus, program one or two screens for tandem or opposing eye movements and stimulate the left and right eye independently. This system can significantly reduce the programming development time needed to conduct an oculomotor study. The VisualEyes System will be discussed in three parts: 1) the oculomotor recording device to acquire eye movement responses, 2) the VisualEyes software written in LabView, to generate an array of stimuli and store responses as text files and 3) offline data analysis. Eye movements can be recorded by several types of instrumentation such as: a limbus tracking system, a sclera search coil, or a video image system. Typical eye movement stimuli such as saccadic steps, vergent ramps and vergent steps with the corresponding responses will be shown. In this video report, we demonstrate the flexibility of a system to create numerous visual stimuli and record eye movements that can be utilized by basic scientists and clinicians to study healthy as well as clinical populations.

Neural control and cognitive processes can be studied through eye movements. The VisualEyes software allows an operator to program stimuli on two computer screens independently using a simple, custom scripting language. The system can stimulate tandem eye movements (saccades and smooth pursuit) or opposing eye movements (vergence) or any combination.

Eye movement studies have provided a strong foundation forming an understanding of how the brain acquires visual information in both the normal and ...

Neuroscience

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing and the perception of the visual world. The recording of eye-gaze patterns has provided good evidence for the contribution of both the visual context and linguistic/world knowledge to language comprehension. Initial research assessed object-context effects to test&#160;theories of modularity in language processing. In the introduction, we describe how subsequent investigations have taken the role of the wider visual context in language processing as a research topic in its own right, asking questions such as how our visual perception of events and of speakers contributes to&#160;comprehension informed by&#160;comprehenders&#39; experience. Among the examined aspects of the visual context are actions, events, a speaker&#39;s gaze, and emotional facial expressions, as well as spatial object configurations. Following an overview of the eye-tracking method and its different applications, we list the key steps of the methodology in the protocol, illustrating how to successfully use it to study visually-situated language comprehension. A final section presents three sets of representative results and illustrates the benefits and limitations of eye tracking for investigating the interplay between the perception of the visual world and language comprehension.

The present article reviews an eye-tracking methodology for studies on language comprehension. To obtain reliable data, key steps of the protocol must be followed. Among these are the correct set-up of the eye tracker (e.g., ensuring good quality of the eye and head images) and accurate calibration.

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing ...

Eye-tracking to Distinguish Comprehension-based and Oculomotor-based Regressive Eye Movements During Reading

Regressive eye movements are eye movements that move backwards through the text and comprise approximately 10-25% of eye movements during reading. As such, understanding the causes and mechanisms of regressions plays an important role in understanding eye movement behavior. Inhibition of return (IOR) is an oculomotor effect that results in increased latency to return attention to a previously attended target versus a target that was not previously attended. Thus, IOR may affect regressions. This paper describes how to design materials to distinguish between regressions caused by comprehension-related and oculomotor processes; the latter is subject to IOR. The method allows researchers to identify IOR and control the causes of regressions. While the method requires tightly controlled materials and large numbers of participants and materials, it allows researchers to distinguish and control the types of regressions that occur in their reading studies.

The method was designed to investigate the role of inhibition of return (IOR) in regressive eye movements during reading. The focus is on differentiating between regressions triggered as a result of comprehension difficulty versus those triggered from oculomotor error, including the role of IOR in the two types of regressions.

Regressive eye movements are eye movements that move backwards through the text and comprise approximately 10-25% of eye movements during reading. As ...

Using the Visual World Paradigm to Study Sentence Comprehension in Mandarin-Speaking Children with Autism

Sentence comprehension relies on the ability to rapidly integrate different types of linguistic and non-linguistic information. However, there is currently a paucity of research exploring how preschool children with autism understand sentences using different types of cues. The mechanisms underlying sentence comprehension remains largely unclear. The present study presents a protocol to examine the sentence comprehension abilities of preschool children with autism. More specifically, a visual world paradigm of eye-tracking is used to explore the moment-to-moment sentence comprehension in the children. The paradigm has multiple advantages. First, it is sensitive to the time course of sentence comprehension and thus can provide rich information about how sentence comprehension unfolds over time. Second, it requires minimal task and communication demands, so it is ideal for testing children with autism. To further minimize the computational burden of children, the present study measures eye movements that arise as automatic responses to linguistic input rather than measuring eye movements that accompany conscious responses to spoken instructions.

We present a protocol to examine the use of morphological cues during real-time sentence comprehension by children with autism.

Sentence comprehension relies on the ability to rapidly integrate different types of linguistic and non-linguistic information. However, there is ...

Exploring Infant Sensitivity to Visual Language using Eye Tracking and the Preferential Looking Paradigm

We discuss the use of the preferential looking paradigm in eye tracking studies in order to study how infants develop, understand, and attend to the world around them. Eye tracking is a safe and non-invasive way to collect gaze data from infants, and the preferential looking paradigm is simple to design and only requires the infant to be attending to the screen. By simultaneously showing two visual stimuli that differ in one dimension, we can assess whether infants show different looking behavior for either stimulus, thus demonstrating sensitivity to that difference. The challenges in such experimental approaches are that experiments must be kept brief (no more than 10 min) and be carefully controlled such that the two stimuli differ in only one way. The interpretation of null results must also be carefully considered. In this paper, we illustrate a successful example of an infant eye tracking study with a preferential looking paradigm to discover that 6-month-olds are sensitive to linguistic cues in a signed language despite having no prior exposure to signed language, suggesting that infants possess intrinsic or innate sensitivities to these cues.

Eye tracking studies using a preferential looking paradigm can be used to study infants&#39;&#160;emerging understanding of, and attention to, their external visual world.

We discuss the use of the preferential looking paradigm in eye tracking studies in order to study how infants develop, understand, and attend to the world ...

Examining Online Syntactic Processing of Spoken Complex Sentences in Chinese Using Dual-Modal Interference Tasks

Working memory (WM) plays a central role in the comprehension of complex sentences. Its function in the processing of spoken complex sentences is especially evident because spoken complex sentence processing is memory-intensive. The dual-modal interference paradigm has been used to examine how the WM system is involved in complex syntactic processing. This article presents two exemplar experiments involving auditory processing with either intra- or extrasentential interferences. In the first experiment, auditory stimuli [spoken-Chinese relative clause (RC) sentences with two syntactic types: subject-gapped (SRC) vs. object-gapped (ORC)] are interfered with via a visually presented lexical decision task within a sentence and manipulated using three different interference timepoints. In the second experiment, the same auditory stimuli, presented via an auditory window moving technique, are interfered with via a visually presented digital recall task beyond the sentence and manipulated using three digital memory loads. By assessing how the primary task of comprehending the RC sentences is affected by the secondary task, we can tackle the controversial issue concerning Chinese RC processing asymmetry. Our results reveal different patterns of RC processing compared to those reported in previous studies. Experiment 1 manifests no clear RC processing advantages in either SRC or ORC; however, a preference for ORC is observed at the ends of sentences, and a preference for SRC is found at the main verb site. Likewise, Experiment 2 presents a dynamic pattern. Under a no-digit load, SRCs show processing advantages in the RC marker region. However, under higher-digit-load interference, ORCs show processing advantages in the same region. These results lead to the conjecture that no obvious or intrinsic processing asymmetry exists in the processing of Chinese RCs. Using the approach of assessing specific interference during syntactic processing, these experiments demonstrate future research applications that explore the processing metrics of spoken sentences involving working memory.

Here, we present a protocol employing dual-modal interference tasks to examine online processing of spoken-Chinese relative clause sentences. Two exemplar experiments involving auditory processing with intra- and extrasentential interferences are described. The paradigm provides a methodology for addressing the nature of working memory and its effects on sentence processing.

Working memory (WM) plays a central role in the comprehension of complex sentences. Its function in the processing of spoken complex sentences is ...

Loneliness Assuaged: Eye-Tracking an Audience Watching Barrage Videos

Researchers usually theorize media exposure based on assumptions of legacy media. However, a new interactive video viewing format, in this case barrage video where viewers&#8217; comments are overlaid over visual content, challenges past perspectives. This study proposes an activation and match satisfaction model to study viewing behaviors of lonely people and to challenge previous claims. It presents a protocol to examine the mechanism of how loners use barrage videos by combining eye tracking and self-report measures. Eye tracking documents the audience&#39;s conscious and subconscious watching behaviors in real time and allows for inference of the amount of allocated cognitive resources in response to rational and emotional content. The self-report gauges the amount of satisfaction obtained. Overall, results from the measures supported an activation and match satisfaction model regarding loners and their barrage video viewing behaviors. Implications are discussed.

The study proposes an activation-match model to study how loneliness is mitigated when a lonely audience watches barrage videos of rational and emotional appeals. The protocol uses eye tracking to document duration and fixation, accounting for the degree of satisfaction when emotional needs are appeased by content and barrage.

Researchers usually theorize media exposure based on assumptions of legacy media. However, a new interactive video viewing format, in this case barrage ...

Investigating Oculomotor Responses in Visual Duration Perception Tasks

Eye Movements in Visual Duration Perception: Disentangling Stimulus from Time in Predecisional Processes

Eye-tracking methods may allow the online monitoring of cognitive processing during visual duration perception tasks, where participants are asked to estimate, discriminate, or compare time intervals defined by visual events like flashing circles. However, and to our knowledge, attempts to validate this possibility have remained inconclusive so far, and results remain focused on offline behavioral decisions made after stimulus appearance. This paper presents an eye-tracking protocol for exploring the cognitive processes preceding behavioral responses in an interval comparison task, where participants viewed two consecutive intervals and had to decide whether it speeded up (first interval longer than second) or slowed down (second interval longer). 
Our main concern was disentangling oculomotor responses to the visual stimulus itself from correlates of duration related to judgments. To achieve this, we defined three consecutive time windows based on critical events: baseline onset, the onset of the first interval, the onset of the second interval, and the end of the stimulus. We then extracted traditional oculomotor measures for each (number of fixations, pupil size) and focused on time-window-related changes to separate the responses to the visual stimulus from those related to interval comparison per se. As we show in the illustrative results, eye-tracking data showed significant differences that were consistent with behavioral results, raising hypotheses on the mechanisms engaged. This protocol is embryonic and will require many improvements, but it represents an important step forward in the current state of art.

Author Spotlight: Exploring the Link Between Time Perception of Visual Stimuli and Reading Skills

We present a protocol that employs eye tracking to monitor eye movements during an interval comparison (duration perception) task based on visual events. The aim is to provide a preliminary guide to separate oculomotor responses to duration perception tasks (comparison or discrimination of time intervals) from responses to the stimulus itself.

Eye-tracking methods may allow the online monitoring of cognitive processing during visual duration perception tasks, where participants are asked to ...

Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

Samenvatting

Meer video's verkennen

Hoofdstukken in deze video