The development of this methodology was supported by the National Science Foundation (NSF #0835854), the Leading House Technologies for Vocation Education, funded by the Swiss State Secretariat for Education, Research and Innovation, and the Harvard School of Education&#39;s Dean Venture Fund.

1. Participant Screening
<ol>
	<li>Ensure that participants with normal or corrected-to-normal vision are recruited. Because participants will be asked to wear a mobile eye-tracker, they can wear contact lenses but not regular eyeglasses.</li>
</ol>
2. Preparation for the Experiment
<ol>
	<li>Eye-tracking devices
	<ol>
		<li>Use any mobile eye-tracker capable of capturing eye movement in real world environments. 
		NOTE: The mobile eye-trackers used here were two Tobii Pro Glasses 2 (see Table of Materials). In addition to specialized cameras that can track eye movements, the glasses are also equipped with an HD scene camera and a microphone so that the gaze can be visualized in the context of the user&#39;s visual field. These glasses capture gaze data 50 times per second. Other researchers have used ASL Mobile Eye5, SMI6, or Pupil-labs7, all of which provide video streams from the scene camera and eye-tracking coordinates at varying sampling rates (30&#8211;120 Hz). The procedure below might vary slightly with other eye-tracking devices.</li>
	</ol>
	</li>
	<li>Fiducial Markers
	<ol>
		<li>The two steps below (i.e., temporal and spatial alignments) require the use of fiducial markers. There are several computer vision libraries that provide researchers with these markers and algorithms to detect them on an image or video feed. The protocol described uses the Chilitag library8.</li>
	</ol>
	</li>
	<li>Temporal alignment
	<ol>
		<li>Because the eye-tracking data are recorded on two separate units, ensure that the data are properly synchronized (Figure 1). Two main methods can be used. This manuscript only covers the first method, because server synchronization works differently with each brand of mobile eye-tracker.
		<ol>
			<li>Briefly show a fiducial marker on a computer screen to mark the beginning and the end of a session. This is similar to a visual &#34;hand clap&#34; (Figure 2).</li>
			<li>Alternatively, use a server to synchronize the clocks of the two data collection units. This method is slightly more accurate and recommended if a higher temporal accuracy is required.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Spatial alignment
	<ol>
		<li>To find if two participants are looking at the same place at the same time, map their gazes to a common plane. This plane can be a picture of the experimental setting (see the left side of Figure 3). Carefully design this image before the experiment.</li>
		<li>Size of the fiducial markers: The general size of the fiducial markers depends on the algorithm used to detect them from the eye-tracking video. Surfaces close to the participants can have smaller fiducial markers, while surfaces further away from them need to be larger, so that they look similar from the participants&#39; perspective. Try different sizes beforehand to make sure that they can be detected from the eye-tracking video.</li>
		<li>Number of fiducial markers: To make the process of mapping gaze points into a common plane successful, make sure to have several fiducial markers visible from the participants&#39; point of view at any given time.</li>
		<li>Location of the fiducial markers: Frame relevant areas of interest with strips of fiducial markers (e.g., see the laptop screen on Figure 3).</li>
	</ol>
	</li>
	<li>Finally, run pilots to test the synchronization procedure and determine the optimal location, size, and number of fiducial markers. Eye-tracking videos can be processed through a computer vision algorithm to see if the fiducial markers are reliably detected.</li>
</ol>
3. Running the experiment
<ol>
	<li>Instructions
	<ol>
		<li>Instruct participants to put on the eye-tracking glasses as they would a normal pair of glasses. Based on the participants&#39; distinct facial features, nose pieces of different heights may need to be used to preserve data quality.</li>
		<li>After turning on the eye-tracker, have the participants clip the recording unit to themselves to allow for natural body movement.</li>
	</ol>
	</li>
	<li>Calibration
	<ol>
		<li>Instruct the participants to look at the center of the calibration marker provided by Tobii while the calibration function of the software is enabled. Once the calibration is complete, recording can be started from within the software.</li>
		<li>Instruct participants to not move the mobile eye-trackers after calibration. If they do, the data are likely to be inaccurate and the calibration procedure will need to be performed again.</li>
	</ol>
	</li>
	<li>Data monitoring
	<ol>
		<li>Monitor the data collection process during the study and ensure that the eye-tracking data are being collected properly. Most mobile eye-trackers can provide a live stream on a separate device (e.g., a tablet) for this purpose.</li>
	</ol>
	</li>
	<li>Data export
	<ol>
		<li>After the recording session is complete, instruct the participant to remove the eye-tracking glasses and the data collection unit. Turn off the unit.</li>
		<li>Extract data using another software, Tobii Pro Lab, by removing the SD card from the data collection unit importing the session data. Tobii Pro Lab can be used to replay the video, create visualizations, and export the eye-tracking data as comma-separated (.csv) or tab-separated (.tsv) files.</li>
	</ol>
	</li>
</ol>
4. Preprocessing the dual eye-tracking data
<ol>
	<li>Sanity checking eye-tracking data
	<ol>
		<li>Check the eye-tracking data visually after data collection. It is not uncommon for some participants to have missing data. For example, some particular eye physiology can pose problems to eye-tracking algorithms, the glasses might shift during the experiment, the data collection software might crash, etc.</li>
		<li>Use descriptive statistics to check how much data were lost during each session and exclude sessions that have significant amounts of missing or noisy data.</li>
	</ol>
	</li>
	<li>Temporal alignment
	<ol>
		<li>Trim the data from each mobile eye-tracker to only include interactions between the participants. This can be achieved by using the method described above (i.e., presenting two special fiducial markers to participants at the start and the end of the session). These fiducial markers can then be detected from the eye-tracking video to trim the datasets.</li>
	</ol>
	</li>
	<li>Spatial alignment 
	NOTE: To detect whether two participants are looking at the same place at the same time, it is necessary to remap the participants&#39; gaze onto a common plane (i.e., an image of the experimental setting). A computational method for achieving this goal is a homography (i.e., a perspective transformation of a plane). From a technical perspective, two images of the same planar surface in space are related by a homography matrix. Based on a common set of points, this matrix can be used to infer the location of additional points between two planes. In Figure 3, for example, if a computer vision algorithm knows where the fiducial markers are on the handout, it can remap the gaze of the participant onto the common plane on the left side. The white lines connect the two sets of points shared by the video feed of each participant and the scene, which are then used for building the homography to remap the green and blue dots on the left side.
	<ol>
		<li>Use the Python version of OpenCV, e.g., to compute the homography matrix from the fiducial markers and then to remap the eye-tracking data to the scene of the experimental setting (or any other suitable library in your language of choice). OpenCV provides two useful functions: findHomography() to get the homography matrix, and perspectiveTransform() to transform the point from one perspective to the other.</li>
		<li>To use findHomography(), run with two arguments: the X,Y coordinates of the source points (i.e., the fiducial markers detected from the participants&#39; scene video, shown on the right in Figure 3) and the corresponding destination points (i.e., the same fiducials markers detected on the scene image, shown on the left in Figure 3).</li>
		<li>Feed the resulting homography matrix into the perspectiveTransform() function, along with a new point that needs to be mapped from the source image to the destination image (e.g., the eye-tracking data shown as a blue/green dot on the right side of Figure 3). The perspectiveTransform function returns the new coordinate of the same point on the scene image (i.e., the blue/green dots shown on the left side of Figure 3). 
		NOTE: For more information, the OpenCV official documentation provides sample code and examples to implement the homography: docs.opencv.org/master/d1/de0/tutorial_py_feature_homography.html.</li>
	</ol>
	</li>
	<li>Sanity checking the homography
	<ol>
		<li>Complete section 4.3 for the entire session, and perform a homography on each frame of the mobile eye-tracking video to check the quality of the homography. While there are no automated ways to estimate the accuracy of the resulting eye-tracking data, videos like the one shown in Figure 4 should be used to manually sanity check each session.</li>
		<li>If the quality is lower than expected, consider additional parameters to improve the results of the homography:
		<ol>
			<li>Number of fiducial markers detected: Only perform the homography if enough fiducial markers can be detected from the video stream. This number can be determined by examining the video produced above.</li>
			<li>Location of the fiducial markers: If different markers are at different depths and orientations, the quality of the homography usually increases when the markers closest to the gaze coordinates are selected, given that there are enough markers to build a robust homography.</li>
			<li>Orientation of the fiducial markers: Combining fiducial markers that have different orientations (e.g., horizontal and vertical) will produce inaccurate homographies. It is recommended to first detect which plane or areas of interests (AOIs) the participant is looking at (e.g., the computer screen, the cheat sheet, the table, see Figure 3) and then use the fiducial markers on this plane for the homography.</li>
			<li>Quality of the video stream: Sudden head movements can blur video frames and make the data unusable, because fiducial markers cannot be reliably detected (Figure 4). The methodology of this paper is not appropriate for experiments that involve a lot of sudden head movements.</li>
		</ol>
		</li>
	</ol>
	</li>
</ol>
5. Analyzing the dual eye-tracking data
<ol>
	<li>Missing data
	<ol>
		<li>In order to make sure that the data were properly remapped onto the reference image, produce visualization graphs (e.g., Figure 5, Figure 6) and descriptive statistics to check how much data are missing.</li>
	</ol>
	</li>
	<li>Cross-recurrence graphs
	<ol>
		<li>Use cross-recurrence graphs9 to represent visual synchronization between two participants (Figure 6), where the X-axis represents time for the first participant, and the Y-axis represents time for the second participant. Black squares indicate that participants are looking at the same area, a black diagonal line describes two subjects looking at the same thing at exactly the same time, and black squares off the diagonal line describes when two subjects looking at the same thing with a time lag. Finally, differentiating between missing data (white square) and existing data with no JVA (gray squares) helps identify problematic sessions. This provides researchers with a visual sanity check.</li>
	</ol>
	</li>
	<li>Computing JVA
	<ol>
		<li>After filtering for missing data, compute a metric for JVA by counting the number of times that participants&#39; gazes are in the same radius in the scene (defined below) in a -2/+2 s time window. Divide this number by the number of valid data points that can be used to compute JVA. The result of the division represents the percentage of time that two subjects were jointly looking at the same place. This last step is necessary to avoid inflating the scores of groups with more data after the homography. 
		NOTE: Two parameters need to be set before JVA can be computed, the minimal distance between two gaze points, and the time window between them (Figure 7): 1) Time window: An early foundational study10 used a single eye-tracker to measure JVA between a listener and a speaker. The researchers asked a first set of participants (&#34;speakers&#34;) to talk about a television show whose characters were displayed in front of them. A second set of participants (&#34;listeners&#34;) then watched the same show while listening to audio recording of the speakers. The eye movements of the speakers and listeners were compared, and it was found that a listener&#39;s eye movements closely matched a speaker&#39;s eye movement with a delay of 2 s. In subsequent work11 researchers analyzed live dialogues and found that a delay of 3 s best captured moments of JVA. Because each task is unique and might exhibit different time lags, it is also suggested to explore how different time lags affect the results of a given experiment. Overall, it is common to look for JVA in a &#177; 2/3 s time window depending on the experimental task and then explore how different time lags might change the results. 2) Distance between gazes: there is no empirically defined distance between two gazes for them to count as JVA. This distance is dependent on the research questions defined by the researchers. The research questions should inform the size of the targets of interest. In the example seen in Figure 7, a radius of 100 pixels on the scene image (blue/green circles) was chosen for the analysis because it is sufficient to capture when participants are looking at the robot in the maze, as well as at similar user interface elements on the computer screen, which are the two main areas of interest for this experimental task.</li>
	</ol>
	</li>
</ol>

<ol>
	<li>Tomasello, M., Moore, C., Dunham, P. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Joint+attention+as+social+cognition.">Joint attention as social cognition.</a> Joint attention: Its origins and role in development. , 103-130 (1995).</li><li>Mundy, P., Sigman, M., Kasari, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+longitudinal+study+of+joint+attention+and+language+development+in+autistic+children.">A longitudinal study of joint attention and language development in autistic children.</a> Journal of Autism and Developmental Disorders. 20, 115-128 (1990).</li><li>Clark, H. H., Brennan, S. E., Resnick, L. B., Levine, J. M., Teasley, S. D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Grounding+in+communication.">Grounding in communication.</a> Perspectives on socially shared cognition. , 127-149 (1991).</li><li>Siposova, B., Carpenter, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+new+look+at+joint+attention+and+common+knowledge.">A new look at joint attention and common knowledge.</a> Cognition. 189, 260-274 (2019).</li><li>Gergle, D., Clark, A. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=See+What+I'm+Saying?:+Using+Dyadic+Mobile+Eye+Tracking+to+Study+Collaborative+Reference.">See What I'm Saying?: Using Dyadic Mobile Eye Tracking to Study Collaborative Reference.</a> Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work. , 435-444 (2011).</li><li>Renner, P., Pfeiffer, T., Wachsmuth, I., Freksa, C., Nebel, B., Hegarty, M., Barkowsky, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Spatial+References+with+Gaze+and+Pointing+in+Shared+Space+of+Humans+and+Robots.">Spatial References with Gaze and Pointing in Shared Space of Humans and Robots.</a> Spatial Cognition IX. , 121-136 (2014).</li><li>Shvarts, A. Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+detection+of+gaze+convergence+in+multimodal+collaboration:+a+dual+eye-tracking+technology.">Automatic detection of gaze convergence in multimodal collaboration: a dual eye-tracking technology.</a> The Russian Journal of Cognitive Science. 5, 4 (2018).</li><li>. Chilitags: Robust Fiducial Markers for Augmented Reality [software] Available from: <a target="_blank" href="https://github.com/chili-epfl/qml-chilitags">https://github.com/chili-epfl/qml-chilitags</a> (2013)</li><li>Jermann, P., Mullins, D., N&#252;ssli, M. -. A., Dillenbourg, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Collaborative+Gaze+Footprints:+Correlates+of+Interaction+Quality.+Connecting+Computer-Supported+Collaborative+Learning+to+Policy+and+Practice.">Collaborative Gaze Footprints: Correlates of Interaction Quality. Connecting Computer-Supported Collaborative Learning to Policy and Practice.</a> CSCL2011 Conference Proceedings., Volume I - Long Papers. , 184-191 (2011).</li><li>Richardson, D. C., Dale, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Looking+To+Understand:+The+Coupling+Between+Speakers'+and+Listeners'+Eye+Movements+and+Its+Relationship+to+Discourse+Comprehension.">Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension.</a> Trends in Cognitive Sciences. 29, 1045-1060 (2005).</li><li>Richardson, D. C., Dale, R., Kirkham, N. Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+Art+of+Conversation+Is+Coordination+Common+Ground+and+the+Coupling+of+Eye+Movements+During+Dialogue.">The Art of Conversation Is Coordination Common Ground and the Coupling of Eye Movements During Dialogue.</a> Psychological Science. 18, 407-413 (2007).</li><li>Schneider, B., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Using+Mobile+Eye-Trackers+to+Unpack+the+Perceptual+Benefits+of+a+Tangible+User+Interface+for+Collaborative+Learning.">Using Mobile Eye-Trackers to Unpack the Perceptual Benefits of a Tangible User Interface for Collaborative Learning.</a> ACM Transactions on Computer-Human Interaction. 23, 1-23 (2016).</li><li>Meier, A., Spada, H., Rummel, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+rating+scheme+for+assessing+the+quality+of+computer-supported+collaboration+processes.">A rating scheme for assessing the quality of computer-supported collaboration processes.</a> Int. J. Comput.-Support. Collab. Learn. 2, 63-86 (2007).</li><li>Schneider, B., Pea, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Real-time+mutual+gaze+perception+enhances+collaborative+learning+and+collaboration+quality.">Real-time mutual gaze perception enhances collaborative learning and collaboration quality.</a> Journal of Computer-Supported Collaborative Learning. 8, 375-397 (2013).</li></ol>

The authors declare that they have no competing financial interests.

The methodology described in this paper provides a rigorous way to capture JVA in colocated dyads. With the emergence of affordable sensing technology and improved computer vision algorithms, it is now possible to study collaborative interactions with an accuracy that was previously unavailable. This methodology leverages fiducial markers disseminated in the environment and uses homographies as a way to remap participants&#39; gazes onto a common plane. This allows researchers to rigorously study JVA in colocated groups.
This method includes multiple sanity checks that need to be performed at various point of the experiment. Because this is a complex procedure, researchers need to make sure that the resulting datasets are complete and valid. Finally, it is recommended to conduct pilot studies before the actual experiment, and to reconstruct participants&#39; interactions though a video after data collection is completed (Figure 3, Figure 4, Figure 5, Figure 6).
There are several limitations associated with this method:
Number of participants. While this methodology works well for two participants, analysis become more complicated with larger groups. Fiducial markers can still be used to remap gazes onto a ground truth but knowing how to identify JVA becomes a more nuanced process. Should JVA be defined as the times when everyone is looking at the same place at the same time, or when two participants are gazing at the same place? Additionally, visualizations like the cross-recurrence graph become impractical with more than 2&#8211;3 people.
Settings. The method described in this paper is appropriate for small, controlled settings (e.g., laboratory studies). Open-ended settings, such as outdoors or large spaces, are usually too complicated to instrument with fiducial markers and thus can limit the usefulness of the eye-tracking data. Additionally, the fiducial markers can be distracting and clutter the environment. In the future, better computer vision algorithms will be able to automatically extract common features between two perspectives. There are already algorithms that exist for this purpose, but we found that the level of accuracy was not yet acceptable for the type of experiment described above.
AOIs. Related to the point above, computing homography and the cross-recurrence graph work well with a stable number of areas of interest, but corrections have to be made when comparing different tasks with different numbers of areas of interest.
Use of equipment. Mobile eye-trackers can be obtrusive, affecting participants&#39; behavior or failing to work with particular eye physiology.
In conclusion, the methodology described in this paper is a promising way to study colocated interactions. It allows researchers to capture a precise metric for JVA, which is a critical construct in the social sciences1. Additionally, it is possible to detect more fine-grained indicators of collaborative learning through this methodology12 compared to traditional qualitative analyses. In short, it is a more efficient and accurate way to study social interactions.
Potential application of this method includes designing interventions to support collaboration through real-time eye-tracking data. Some pioneering work has produced shared gaze visualizations using remote eye-trackers, which has been shown to benefit collaborative learning from a distance14. Dyads who could see the gaze of their partner in real time exhibited more JVA, collaborated better and achieved higher learning gains compared to a control group. Future work will examine whether this kind of intervention can support collaborative processes in colocated settings (e.g., through virtual or augmented reality headsets).

JVA has been extensively studied over the last century, especially by developmental psychologists studying language acquisition. It was quickly established that joint attention is more than just a way to learn words but rather a precursor to children&#39;s theories of mind1. Thus, it plays a significant role in many social processes, such as communicating with others, collaborating, and developing empathy. Autistic children, for instance, lack the ability to coordinate their visual attention with their caregivers, which is associated with significant social impairments2. Humans need joint attention to become functional members of society, to coordinate their actions, and to learn from others. From children acquiring their first words, teenagers learning from schoolteachers, students collaborating on projects, and to groups of adults working toward common goals, joint attention is a fundamental mechanism to establish common ground between individuals3. In this paper, I focus on the study of JVA in educational research. Understanding how joint attention unfolds over time is of primary importance for the study of collaborative learning processes. As such, it plays a predominant role in socioconstructivist settings.
The exact definition of joint attention is still debated4. This paper is concerned with a subconstruct of joint attention (JA), namely JVA. JVA happens when two subjects are looking at the same place at the same time. It should be noted that JVA does not provide any information about other important constructs of interest in the study of JA, such as monitoring common, mutual, and shared attention, or more generally, awareness of the cognition of another group member. This paper operationalizes and simplifies JVA by combining the eye-tracking data from two participants and analyzing the frequency in which they align their gazes. For a more comprehensive discussion, the interested reader can learn more about the study of the JA construct in Siposovaet al.4.
Over the past decade, technological advances have radically transformed research on JVA. The main paradigm shift was to use multiple eye-trackers to obtain quantitative measures of attentional alignments, as opposed to qualitatively analyzing video recordings in a laboratory or ecological setting. This development has allowed researchers to collect precise, detailed information about dyads&#39; visual coordination. Additionally, eye-trackers are becoming more affordable: until recently, their use was reserved to academic settings or large corporations. It is now possible to purchase inexpensive eye-trackers that generate reliable datasets. Finally, the progressive inclusion of gaze-tracking capabilities into existing devices like high-end laptops and virtual and augmented reality headsets suggests that eye-tracking will soon become ubiquitous.
Because of the popularization of eye-tracking devices, it is important to understand what they can and cannot tell us about social interactions. The methodology presented in this paper marks a first step in this direction. I address two challenges in capturing JVA from multiple eye-trackers: synchronizing the data on 1) the temporal scale, and 2) on the spatial scale. More specifically, this protocol makes use of fiducial markers placed in real-world environments to inform computer vision algorithms where participants are orienting their gaze. This new kind of methodology paves the way to rigorous analysis of human behavior in small groups.
This research protocol complies with the guidelines of Harvard University&#39;s human research ethics committee.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Tobii Glasses 2</td>
			<td>Tobii</td>
			<td>N/A</td>
			<td><a href="https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/" target="_blank">https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/</a></td>
		</tr>
		<tr>
			<td>Fiducial markers</td>
			<td>Chili lab &#8211; EPFL, Switzerland</td>
			<td>N/A</td>
			<td><a href="https://github.com/chili-epfl/chilitags" target="_blank">https://github.com/chili-epfl/chilitags</a></td>
		</tr>
	</tbody>
</table>

a methodology for capturing joint visual attention using mobile eye-trackers

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency sensors, such as eye-trackers, electrodermal activity wristbands, EEG bands, and motion sensors provide observations at the millisecond level. This level of precision allows researchers to collect large datasets on social interactions. In this paper, I discuss how multiple eye-trackers can capture a fundamental construct in social interactions, joint visual attention (JVA). JVA has been studied by developmental psychologists to understand how children acquire language, learning scientists to understand how small groups of learners work together, and social scientists to understand interactions in small teams. This paper describes a methodology for capturing JVA in colocated settings using mobile eye-trackers. It presents some empirical results and discusses implications of capturing microobservations to understand social interactions.

The methodology presented above was used to study students who were following a vocational training program in logistics (n = 54)12. In this experiment, pairs of students interacted with a Tangible User Interface (TUI) that simulated a small-scale warehouse. The fiducial markers placed on the TUI allowed the research team to remap students&#39; gazes onto a common plane and compute levels of JVA. Findings indicated that groups who had higher levels of JVA tended to do better at the task given to them, learned more, and had a better quality of collaboration13 (Figure 8, left side). Dual eye-tracking datasets also allowed us to capture particular group dynamics like the free-rider effect. We estimated this effect by identifying who was likely to have initiated each moment of JVA (i.e., whose gaze was there first) and who responded to it (i.e., whose gaze was there second). We found a significant correlation between learning gains and the students&#39; tendency to equally share the responsibility of initiating and responding to offers of JVA. In other words, groups in which the same person always initiated moments of JVA were less likely to learn (Figure 8, right side) and groups where this responsibility was equally shared were more likely to learn. This finding shows that we can go beyond merely quantifying JV, and actually identify group dynamics and productivity using dual eye-tracking data.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/60670/60670fig1.jpg" /> 
Figure 1: Each participant generates two video feeds with the X,Y coordinates of their gaze on each video frame. This methodology addresses synchronizing the data temporally and spatially between the participants. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/60670/60670fig1alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/60670/60670fig2.jpg" /> 
Figure 2: A methodology for synchronizing the two datasets. Briefly showing unique fiducial marker on a computer screen to tag the start and the end of the activity. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/60670/60670fig2alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/60670/60670fig3.jpg" /> 
Figure 3: Using fiducial markers disseminated in the environment to remap participants&#39; gazes onto a common plan (left side). White lines indicate fiducial markers that have been detected in both images. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/60670/60670fig3alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/60670/60670fig4.jpg" /> 
Figure 4: Examples of poor data quality. Left: A blurred frame from the eye-tracking video caused by a sudden head movement. Fiducial markers could not be detected in this image. Right: A failed homography where the fiducial marker data were not properly synchronized with the video feed. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/60670/60670fig4alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 5" class="xfigimg" src="/files/ftp_upload/60670/60670fig5.jpg" /> 
Figure 5: Heatmaps. Left: A heatmap of the eye-tracking data remapped onto the experimental scene. This visualization was used as a sanity check for the homography. Right: A group that had too much missing data and had to be discarded. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/60670/60670fig5alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 6" class="xfigimg" src="/files/ftp_upload/60670/60670fig6.jpg" /> 
Figure 6: Cross recurrence graph generated from three dyads to visualize JVA. P1 represents time for the first participant, P2 represents time for the second participant. Black squares show JVA; gray squares show moments where participants are looking at different places; white squares show missing data. Square along the main diagonal indicates moments where participants looked at the same place at the same time. This visualization was used as a sanity check for measures of JVA from the combined eye-tracking data. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/60670/60670fig6alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 7" class="xfigimg" src="/files/ftp_upload/60670/60670fig7.jpg" /> 
Figure 7: A video frame where JVA was detected between two participants (red dots). Richardson et al.11 recommend looking at a time window of +/-2 s. when computing JVA. Additionally, researchers need to define the minimal distance between two gaze points to count as JVA. A radius of 100 pixel was chosen on the middle image above. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/60670/60670fig7large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 8" class="xfigimg" src="/files/ftp_upload/60670/60670fig8.jpg" /> 
Figure 8: Examples of results. Data from Schneider et al.12 where the percentage of time looking at the same place at the same time was correlated with participants&#39; quality of collaboration: r(24) = 0.460, P = 0.018 (left side) and imbalances in initiating/responding to offers of JVA was correlated with their learning gains: r(24) = &#8722;0.47, P = 0.02 (right side). <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/60670/60670fig8alarge.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

Watch this Scientific Journal Video about A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers at JoVE.com

A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers

Using multimodal sensors is a promising way to understand the role of social interactions in educational settings. This paper describes a methodology for capturing joint visual attention from colocated dyads using mobile eye-trackers.

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency ...

a-methodology-for-capturing-joint-visual-attention-using-mobile-eye

Nanyang Technological University

Research

JoVE Journal

Behavior

7.5K Views.  Harvard University. Using multimodal sensors is a promising way to understand the role of social interactions in educational settings. This paper describes a methodology for capturing joint visual attention from colocated dyads using mobile eye-trackers.

Video: A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers

Method for Simultaneous fMRI/EEG Data Collection during a Focused Attention Suggestion for Differential Thermal Sensation

In the present work, we demonstrate a method for concurrent collection of EEG/fMRI data. In our setup, EEG data are collected using a high-density 256-channel sensor net. The EEG amplifier itself is contained in a field isolation containment system (FICS), and MRI clock signals are synchronized with EEG data collection for subsequent MR artifact characterization and removal. We demonstrate this method first for resting state data collection. Thereafter, we demonstrate a protocol for EEG/fMRI data recording, while subjects listen to a tape asking them to visualize that their left hand is immersed in a cold-water bath and referred to, here, as the cold glove paradigm. Thermal differentials between each hand are measured throughout EEG/fMRI data collection using an MR compatible temperature sensor that we developed for this purpose. We collect cold glove EEG/fMRI data along with simultaneous differential hand temperature measurements both before and after hypnotic induction. Between pre and post sessions, single modality EEG data are collected during the hypnotic induction and depth assessment process. Our representative results demonstrate that significant changes in the EEG power spectrum can be measured during hypnotic induction, and that hand temperature changes during the cold glove paradigm can be detected rapidly using our MR compatible differential thermometry device.

We present a protocol for concurrent collection of EEG/fMRI data, and synchronized MR clock signal recording. We demonstrate this method using a unique paradigm whereby subjects receive &#x2018;cold glove&#x2019; instructions during scanning, and EEG/fMRI data are recorded along with hand temperature measurements both before and after hypnotic induction.

In the present work, we demonstrate a method for concurrent collection of EEG/fMRI data. In our setup, EEG data are collected using a high-density ...

Automated Visual Cognitive Tasks for Recording Neural Activity Using a Floor Projection Maze

Neuropsychological tasks used in primates to investigate mechanisms of learning and memory are typically visually guided cognitive tasks. We have developed visual cognitive tasks for rats using the Floor Projection Maze1,2&#160;that are optimized for visual abilities of rats permitting stronger comparisons of experimental findings with other species.
In order to investigate neural correlates of learning and memory, we have integrated electrophysiological recordings into fully automated cognitive tasks on the Floor Projection Maze1,2. Behavioral software interfaced with an animal tracking system allows monitoring of the animal's behavior with precise control of image presentation and reward contingencies for better trained animals. Integration with an in vivo electrophysiological recording system enables examination of behavioral correlates of neural activity at selected epochs of a given cognitive task.
We describe protocols for a model system that combines automated visual presentation of information to rodents and intracranial reward with electrophysiological approaches. Our model system offers a sophisticated set of tools as a framework for other cognitive tasks to better isolate and identify specific mechanisms contributing to particular cognitive processes.

We describe protocols for training rats for chronic electrophysiological recordings in fully automated cognitive tasks on a Floor Projection Maze.

Neuropsychological tasks used in primates to investigate mechanisms of learning and memory are typically visually guided cognitive tasks. We have ...

The 5-Choice Serial Reaction Time Task: A Task of Attention and Impulse Control for Rodents

This protocol describes the 5-choice serial reaction time task, which is an operant based task used to study attention and impulse control in rodents. Test day challenges, modifications to the standard task, can be used to systematically tax the neural systems controlling either attention or impulse control. Importantly, these challenges have consistent effects on behavior across laboratories in intact animals and can reveal either enhancements or deficits in cognitive function that are not apparent when rats are only tested on the standard task. The variety of behavioral measures that are collected can be used to determine if other factors (i.e., sedation, motivation deficits, locomotor impairments) are contributing to changes in performance. The versatility of the 5CSRTT is further enhanced because it is amenable to combination with pharmacological, molecular, and genetic techniques.

This protocol describes the 5-choice serial reaction time task, which is an operant based task used to study attention and impulse control in rodents. Test day challenges, which are modifications of the standard task, increase flexibility of the task and can be combined with other manipulations to more fully characterize behavior.

This protocol describes the 5-choice serial reaction time task, which is an operant based task used to study attention and impulse control in rodents. ...

Topographical Estimation of Visual Population Receptive Fields by fMRI

Visual cortex is retinotopically organized so that neighboring populations of cells map to neighboring parts of the visual field. Functional magnetic resonance imaging allows us to estimate voxel-based population receptive fields (pRF), i.e., the part of the visual field that activates the cells within each voxel. Prior, direct, pRF estimation methods1 suffer from certain limitations: 1) the pRF model is chosen a-priori and may not fully capture the actual pRF shape, and 2) pRF centers are prone to mislocalization near the border of the stimulus space. Here a new topographical pRF estimation method2 is proposed that largely circumvents these limitations. A linear model is used to predict the Blood Oxygen Level-Dependent (BOLD) signal by convolving the linear response of the pRF to the visual stimulus with the canonical hemodynamic response function. PRF topography is represented as a weight vector whose components represent the strength of the aggregate response of voxel neurons to stimuli presented at different visual field locations. The resulting linear equations can be solved for the pRF weight vector using ridge regression3, yielding the pRF topography. A pRF model that is matched to the estimated topography can then be chosen post-hoc, thereby improving the estimates of pRF parameters such as pRF-center location, pRF orientation, size, etc. Having the pRF topography available also allows the visual verification of pRF parameter estimates allowing the extraction of various pRF properties without having to make a-priori assumptions about the pRF structure. This approach promises to be particularly useful for investigating the pRF organization of patients with disorders of the visual system.

It is important to obtain unbiased estimates of visual population receptive fields (pRFs) by functional magnetic resonance imaging. We use mild regularization constraints to estimate pRF topography without a-priori assumptions about pRF shape, allowing us to choose specific pRF models post-hoc. This is particularly advantageous in subjects with visual-pathway lesions.

Visual cortex is retinotopically organized so that neighboring populations of cells map to neighboring parts of the visual field. Functional magnetic ...

Methods to Test Visual Attention Online

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative data samples than can typically be collected on college campuses. However, before such methods can be widely adopted, a number of technological challenges must be overcome &#8211; in particular in experiments where tight control over stimulus properties is necessary. Here we present methods for collecting performance data on two tests of visual attention. Both tests require control over the visual angle of the stimuli (which in turn requires knowledge of the viewing distance, monitor size, screen resolution, etc.) and the timing of the stimuli (as the tests involve either briefly flashed stimuli or stimuli that move at specific rates). Data collected on these tests from over 1,700 online participants were consistent with data collected in laboratory-based versions of the exact same tests. These results suggest that with proper care, timing/stimulus size dependent tasks can be deployed in web-based settings.

To replicate laboratory settings, online data collection methods for visual tasks require tight control over stimulus presentation. We outline methods for the use of a web application to collect performance data on two tests of visual attention.

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative ...

Visualizing Visual Adaptation

Many techniques have been developed to visualize how an image would appear to an individual with a different visual sensitivity: e.g., because of optical or age differences, or a color deficiency or disease. This protocol describes a technique for incorporating sensory adaptation into the simulations. The protocol is illustrated with the example of color vision, but is generally applicable to any form of visual adaptation. The protocol uses a simple model of human color vision based on standard and plausible assumptions about the retinal and cortical mechanisms encoding color and how these adjust their sensitivity to both the average color and range of color in the prevailing stimulus. The gains of the mechanisms are adapted so that their mean response under one context is equated for a different context. The simulations help reveal the theoretical limits of adaptation and generate &#34;adapted images&#34; that are optimally matched to a specific environment or observer. They also provide a common metric for exploring the effects of adaptation within different observers or different environments. Characterizing visual perception and performance with these images provides a novel tool for studying the functions and consequences of long-term adaptation in vision or other sensory systems.

This article describes a novel method for simulating and studying adaptation in the visual system.

Many techniques have been developed to visualize how an image would appear to an individual with a different visual sensitivity: e.g., because of optical ...

Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The proposed method is based on a new and synergistic combination of three components: the temporal-order judgments (TOJ) paradigm, Bundesen&#39;s Theory of Visual Attention (TVA), and a hierarchical Bayesian estimation framework. The method provides readily interpretable parameters, which are supported by the theoretical and neurophysiological underpinnings of TVA. Using TOJs, TVA-based estimates can be obtained for a broad range of stimuli, whereas traditional paradigms used with TVA are mainly limited to letters and digits. Finally, the meaningful parameters of the proposed model allow for the establishment of a hierarchical Bayesian model. Such a statistical model allows assessing results in one coherent analysis both on the subject and the group level.
To demonstrate the feasibility and versatility of this new approach, three experiments are reported with attention manipulations in synthetic pop-out displays, natural images, and a cued letter-report paradigm.

Temporal-order judgments can be used to estimate processing speed parameters and attentional weights and thereby to infer the mechanisms of attentional processing. This methodology can be applied to a wide range of visual stimuli and works with many attention manipulations.

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The ...

Central and Divided Visual Field Presentation of Emotional Images to Measure Hemispheric Differences in Motivated Attention

Two dominant theories on lateralized processing of emotional information exist in the literature. One theory posits that unpleasant emotions are processed by right frontal regions, while pleasant emotions are processed by left frontal regions. The other theory posits that the right hemisphere is more specialized for the processing of emotional information overall, particularly in posterior regions.
Assessing the different roles of the cerebral hemispheres in processing emotional information can be difficult without the use of neuroimaging methodologies, which are not accessible or affordable to all scientists. Divided visual field presentation of stimuli can allow for the investigation of lateralized processing of information without the use of neuroimaging technology. 
This study compared central versus divided visual field presentations of emotional images to assess differences in motivated attention between the two hemispheres. The late positive potential (LPP) was recorded using electroencephalography (EEG) and event-related potentials (ERPs) methodologies to assess motivated attention. Future work will pair this paradigm with a more active behavioral task to explore the behavioral impacts on the attentional differences found.

This study compared central versus divided visual field presentations of emotional images to assess differences in motivated attention between the two hemispheres. The late positive potential (LPP) was recorded using electroencephalography (EEG) and event-related potentials (ERPs) methodologies to assess motivated attention.

Two dominant theories on lateralized processing of emotional information exist in the literature. One theory posits that unpleasant emotions are processed ...

Methodology for Establishing a Community-Wide Life Laboratory for Capturing Unobtrusive and Continuous Remote Activity and Health Data

An end-to-end suite of technologies has been established for the unobtrusive and continuous monitoring of health and activity changes occurring in the daily life of older adults over extended periods of time. The technology is aggregated into a system that incorporates the principles of being minimally obtrusive, while generating secure, privacy protected, continuous objective data in real-world (home-based) settings for months to years. The system includes passive infrared presence sensors placed throughout the home, door contact sensors installed on exterior doors, connected physiological monitoring devices (such as scales), medication boxes, and wearable actigraphs. Driving sensors are also installed in participants' cars and computer (PC, tablet or smartphone) use is tracked. Data is annotated via frequent online self-report options that provide vital information with regard to the data that is difficult to infer via sensors such as internal states (e.g., pain, mood, loneliness), as well as data referent to activity pattern interpretation (e.g., visitors, rearranged furniture). Algorithms have been developed using the data obtained to identify functional domains key to health or disease activity monitoring, including mobility (e.g., room transitions, steps, gait speed), physiologic function (e.g., weight, body mass index, pulse), sleep behaviors (e.g., sleep time, trips to the bathroom at night), medication adherence (e.g., missed doses), social engagement (e.g., time spent out of home, time couples spend together), and cognitive function (e.g., time on computer, mouse movements, characteristics of online form completion, driving ability). Change detection of these functions provides a sensitive marker for the application in health surveillance of acute illnesses (e.g., viral epidemic) to the early detection of prodromal dementia syndromes. The system is particularly suitable for monitoring the efficacy of clinical interventions in natural history studies of geriatric syndromes and in clinical trials.

Unobtrusive sensors and pervasive computing technology incorporated into the daily home life of older adults enables meaningful health and activity changes to be recorded continuously for months to years, providing ecologically valid, high frequency, multi-domain data for research or clinical use.

An end-to-end suite of technologies has been established for the unobtrusive and continuous monitoring of health and activity changes occurring in the ...

Visual Classical Conditioning in Wood Ants

Several species of insects have become model systems for studying learning and memory formation. Although many studies focus on freely moving animals, studies implementing classical conditioning paradigms with harnessed insects have been important for investigating the exact cues that individuals learn and the neural mechanisms underlying learning and memory formation. Here we present a protocol for evoking visual associative learning in wood ants through classical conditioning. In this paradigm, ants are harnessed and presented with a visual cue (a blue cardboard), the conditional stimulus (CS), paired with an appetitive sugar reward, the unconditional stimulus (US). Ants perform a Maxilla-Labium Extension Reflex (MaLER), the unconditional response (UR), which can be used as a readout for learning. Training consists of 10 trials, separated by a 5-minute intertrial interval (ITI). Ants are also tested for memory retention 10 minutes or 1 hour after training. This protocol has the potential to allow researchers to analyze, in a precise and controlled manner, the details of visual memory formation and the neural basis of learning and memory formation in wood ants.

We present a protocol for the classical conditioning of harnessed ants that permits researchers to study visual learning and memory formation at a level of analysis not possible with freely moving individuals.

Several species of insects have become model systems for studying learning and memory formation. Although many studies focus on freely moving animals, ...