Name: measuring attention and visual processing speed by model-based analysis of temporal-order judgments
Uploaded: 2017-01-23T14:00:00.000+00:00
Duration: 13 min
Description: Watch this Scientific Journal Video about Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments at JoVE.com

Parts of this work have been supported by the German Research Foundation (DFG) via grants 1515/1-2 and 1515/6-1 to Ingrid Scharlau.

NOTE: Some steps in this protocol can be accomplished using custom software provided (along with installation instructions) at http://groups.upb.de/viat/TVATOJ. In the protocol, this collection of programs and scripts is referred to as &#8220;TVATOJ&#8221;.
1. Selection of Stimulus Material
<ol>
	<li>Select stimuli according to the research question. 
	NOTE: In general, two targets are shown at different locations on the screen. Stimuli that have been used with the present method include, for example, shapes, digits, letters, singletons in pop-out displays, and objects in natural images. The latter three types were used in this protocol. 
	NOTE: Several different stimulus types are included in the TOJ plugin (&#34;psylab_toj_stimulus&#34; provided with TVATOJ) for the experiment builder OpenSesame12.</li>
	<li>When creating new stimulus types, make sure that the properties of interest have to be encoded for the judgment by making them important for the task or select stimuli where the properties of interest are encoded automatically (e.g., singletons in pop-out displays).</li>
</ol>
2. Power Estimation and Planning
<ol>
	<li>Perform a Bayesian power analysis by simulating data sets with the selected model, planned design (SOA distribution and repetitions), sample sizes, and hypothesized parameters. Estimate whether it is likely to reach the research goal (for instance, a certain difference in the parameters). If the power is not sufficient, alter the design by adding or shifting SOAs or repetitions and repeat the analysis.
	<ol>
		<li>To use the provided TVATOJ software, open and edit the script &#34;exp1-power.R&#34;. Follow the comments in the file to adjust it for the specific analysis. For general information on Bayesian power estimation refer to Kruschke13.</li>
	</ol>
	</li>
</ol>
3. Specification or Programming of the Experiment
<ol>
	<li>Use an experiment builder or psychophysical presentation library to implement the experiment.
	<ol>
		<li>To use the OpenSesame TOJ plugin provided in TVATOJ, drag the &#34;psylab_toj_stimulus&#34; plugin into a trial presentation loop. Alternatively, open the &#34;simple-toj.osexp&#34; example experiment in OpenSesame.</li>
		<li>Select the desired stimulus type from the dropdown menu &#34;Stimulus type&#34; in the psylab_toj_stimulus configuration. Follow the instructions in TVATOJ for adding new stimulus types if required.</li>
	</ol>
	</li>
	<li>Specify the trials as described in the following steps.
	<ol>
		<li>For every experimental condition, create trials with the planned SOAs. When using the psylab_toj_stimulus plugin and OpenSesame, add all varied factors as variables to the trial loop (e.g. &#34;SOA&#34;).</li>
		<li>Add rows to the table to realize all factor combinations (e.g., seven SOAs, from -100 to 100 msec, crossed with the experimental conditions &#34;attention&#34; and &#34;neutral&#34;). Adjust the loop&#39;s &#34;Repeat&#34; attribute to create sufficient repetitions (see protocol step 2 for determining the distribution and repetition of SOAs). 
		NOTE: Typically, at most 800 trials can be presented within one hour. If more repetitions are needed, consider splitting the experiment into several sessions. Make sure that the &#34;Order&#34; attribute of the loop is set to &#34;Random&#34; before running the experiment.</li>
		<li>In the psylab_toj_stimulus plugin configuration, add placeholders (e.g. &#34;[SOA]&#34;) for the varied factors in the respective fields. Enter constant values in the fields of factors that are not varied. 
		NOTE: Before running the experiment, make sure that accurate timing is guaranteed. If appropriate timing behavior of newer monitors was not verified, use CRT monitors and synchronize with the vertical retrace signal12.</li>
	</ol>
	</li>
</ol>
4. Experimental Procedure
<ol>
	<li>Welcoming and briefing of the participants
	<ol>
		<li>Welcome the participants and inform them about the general form of the experiment (computer-based perception experiment). Inform the participants about the prospective duration of the experiment. Obtain the participants&#39; informed consent to participate in the experiment.</li>
		<li>Ensure that the participants show normal or corrected-to-normal vision (optimally by conducting short vision tests). Some deficits, such as color blindness, may be tolerable if they do not interfere with the research question for the particular type of stimulus material.</li>
		<li>Provide a quiet booth where the experiment is conducted. Adjust the chair, chin rest, keyboard position, and so on, to ensure optimal viewing and response conditions for the experiment.</li>
		<li>Make the participants aware that the experiment requires attention and mental focus and can be fatiguing. Ask them to take short breaks when required. It is, however, equally important not to perform these simple tasks under strong attentional strain. Tell the participants that it is okay to make some errors.</li>
	</ol>
	</li>
	<li>Instruction and warm-up
	<ol>
		<li>Present onscreen instructions for the task, detailing the presentation sequence and response collection procedure. Inform the participants that the task is to report the order in which the targets arrived, and that this will be difficult in some trials. Ask the participants to report their first impression when they cannot tell the order for certain, and let them guess if they have no such impression at all. 
		NOTE: In the binary TOJs used here, there is no option to indicate the perception of simultaneity. To avoid excessive guessing, do not point out the presence of trials with simultaneously presented targets explicitly. Let these simply be difficult trials with the instructions outlined above.</li>
		<li>To avoid eye movements during the trials, ask participants to fixate a mark which is shown in the center of the screen. Ask them to rest their head on a chin rest.</li>
		<li>Ask the participants to take short breaks if necessary. Let them know when breaks are allowed and when they must be avoided (e.g., during the target presentation and before the response).</li>
		<li>Include a short training in which participants can get used to the task. To this end, present a random subset of the experimental trials (see protocol step 3.2). 
		NOTE: Because the task itself is rather simple, ten to twenty trials are usually sufficient. It can be advantageous to increase the participants&#39; confidence in their performance in this task. This can be done by slowing down the presentation and providing feedback.</li>
		<li>Obtain the participants&#39; confirmation that they have understood the task (let them explain it) and that they have no further questions.</li>
	</ol>
	</li>
	<li>Running the main experiment
	<ol>
		<li>Let the experimental software start with the presentation of the main trials. Leave the booth for the main experiment.</li>
	</ol>
	</li>
</ol>
5. Model-based Analysis of the TOJ Data
<ol>
	<li>Convert the raw data files into counts of &#34;probe first&#34; judgments for every SOA. For instance, run the script &#34;os2toj.py&#34; provided with TVATOJ.</li>
	<li>Run the Bayesian estimation procedure to estimate the main parameters wp and C, the derived ones vp and vr and the differences of the parameters. For this purpose, run the script &#34;run-evaluation.R&#34; after editing it according to the instructions provided in the file.</li>
	<li>When the sampling has finished, the differences of interest for the research questions can be assessed. Examples can be found in the following section.</li>
</ol>

<ol>
	<li>Posner, M. I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Orienting+of+attention.">Orienting of attention.</a> Quarterly Journal of Experimental Psychology. 32 (1), 3-25 (1980).</li><li>Van der Heijden, A., Wolters, G., Groep, J., Hagenaar, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Single-letter+recognition+accuracy+benefits+from+advance+cuing+of+location.">Single-letter recognition accuracy benefits from advance cuing of location.</a> Perception &amp; Psychophysics. 42 (5), 503-509 (1987).</li><li>Bundesen, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+theory+of+visual+attention.">A theory of visual attention.</a> Psychological Review. 97 (4), 523-547 (1990).</li><li>Bublak, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Staged+decline+of+visual+processing+capacity+in+mild+cognitive+impairment+and+Alzheimer's+disease.">Staged decline of visual processing capacity in mild cognitive impairment and Alzheimer's disease.</a> Neurobiology of Aging. 32 (7), 1219-1230 (2011).</li><li>Petersen, A., Kyllingsb&#230;k, S., Bundesen, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Measuring+and+modeling+attentional+dwell+time.">Measuring and modeling attentional dwell time.</a> Psychonomic Bulletin &amp; Review. 19 (6), 1029-1046 (2012).</li><li>Vroomen, J., Keetels, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Perception+of+intersensory+synchrony:+A+tutorial+review.">Perception of intersensory synchrony: A tutorial review.</a> Attention, Perception, &amp; Psychophysics. 72 (4), 871-884 (2010).</li><li>T&#252;nnermann, J., Petersen, A., Scharlau, I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Does+attention+speed+up+processing?+Decreases+and+increases+of+processing+rates+in+visual+prior+entry.">Does attention speed up processing? Decreases and increases of processing rates in visual prior entry.</a> Journal of Vision. 15 (3), 1-27 (2015).</li><li>Kr&#252;ger, A., T&#252;nnermann, J., Scharlau, I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fast+and+conspicuous?+Quantifying+salience+with+the+Theory+of+Visual+Attention.">Fast and conspicuous? Quantifying salience with the Theory of Visual Attention.</a> Advances in Cognitive Psychology. 12 (1), 20 (2016).</li><li>Bundesen, C., Habekost, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Principles of Visual Attention: Linking Mind and Brain. , (2008).</li><li>Plummer, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=JAGS:+A+program+for+analysis+of+Bayesian+graphical+models+using+Gibbs+sampling.">JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling.</a> Proceedings of the 3rd international workshop on distributed statistical computing. , 124-125 (2003).</li><li>Kruschke, J. K., Vanpaemel, W., Busemeyer, J., Townsend, J., Wang, Z. J., Eidels, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Bayesian+estimation+in+hierarchical+models.">Bayesian estimation in hierarchical models.</a> The Oxford Handbook of Computational and Mathematical Psychology. , 279-299 (2015).</li><li>Math&#244;t, S., Schreij, D., Theeuwes, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=OpenSesame:+An+open-source,+graphical+experiment+builder+for+the+social+sciences.">OpenSesame: An open-source, graphical experiment builder for the social sciences.</a> Behavior Research Methods. 44 (2), 314-324 (2012).</li><li>Kruschke, J. K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan. , (2015).</li><li>Rensink, R. A., O'Regan, J. K., Clark, J. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=To+see+or+not+to+see:+The+need+for+attention+to+perceive+changes+in+scenes.">To see or not to see: The need for attention to perceive changes in scenes.</a> Psychological Science. 8 (5), 368-373 (1997).</li><li>T&#252;nnermann, J., Kr&#252;ger, N., Mertsching, B., Mustafa, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Affordance+estimation+enhances+artificial+visual+attention:+Evidence+from+a+change-blindness+study.">Affordance estimation enhances artificial visual attention: Evidence from a change-blindness study.</a> Cognitive Computation. 7 (5), 525-538 (2015).</li><li>Shore, D. I., Klein, R. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+effects+of+scene+inversion+on+change+blindness.">The effects of scene inversion on change blindness.</a> The Journal of General Psychology. 127 (1), 27-43 (2000).</li><li>Scharlau, I., Neumann, O. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Temporal+parameters+and+time+course+of+perceptual+latency+priming.">Temporal parameters and time course of perceptual latency priming.</a> Acta Psychologica. 113 (2), 185-203 (2003).</li><li>Schneider, K. A., Bavelier, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Components+of+visual+prior+entry.">Components of visual prior entry.</a> Cognitive Psychology. 47 (4), 333-366 (2003).</li><li>Scharlau, I., Neumann, O. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Perceptual+latency+priming+by+masked+and+unmasked+stimuli:+Evidence+for+an+attentional+interpretation.">Perceptual latency priming by masked and unmasked stimuli: Evidence for an attentional interpretation.</a> Psychological Research. 67 (3), 184-196 (2003).</li><li>Shore, D. I., Spence, C., Klein, R. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Visual+prior+entry.">Visual prior entry.</a> Psychological Science. 12 (3), 205-212 (2001).</li><li>Alcal&#225;-Quintana, R., Garc&#237;a-P&#233;rez, M. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fitting+model-based+psychometric+functions+to+simultaneity+and+temporal-order+judgment+data:+MATLAB+and+R+routines.">Fitting model-based psychometric functions to simultaneity and temporal-order judgment data: MATLAB and R routines.</a> Behavior Research Methods. 45 (4), 972-998 (2013).</li><li>Hoffman, M. D., Gelman, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+No-U-turn+sampler:+adaptively+setting+path+lengths+in+Hamiltonian+Monte+Carlo.">The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo.</a> Journal of Machine Learning Research. 15 (1), 1593-1623 (2014).</li><li>Lee, M. D., Wagenmakers, E. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Bayesian cognitive modeling: A practical course. , (2014).</li><li>Vangkilde, S., Bundesen, C., Coull, J. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Prompt+but+inefficient:+Nicotine+differentially+modulates+discrete+components+of+attention.">Prompt but inefficient: Nicotine differentially modulates discrete components of attention.</a> Psychopharmacology. 218 (4), 667-680 (2011).</li><li>T&#252;nnermann, J., Scharlau, I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Peripheral+Visual+Cues:+Their+Fate+in+Processing+and+Effects+on+Attention+and+Temporal-order.">Peripheral Visual Cues: Their Fate in Processing and Effects on Attention and Temporal-order.</a> Front. Psychol. 7 (1442), (2016).</li></ol>

The authors have nothing to disclose.

The protocol in this article describes how to conduct simple TOJs and fit the data with models based on fundamental stimulus encoding. Three experiments demonstrated how the results can be evaluated in a hierarchical Bayesian estimation framework to assess the influence of attention in highly different stimulus material. Salience in pop-out displays led to increased attentional weights. Also, increased weights were estimated for action space objects in natural images. However, due to the persisting advantage when spatial relations were disturbed by showing such images upside down, it is likely that another local attentional benefit leads to the weight increase. A peripheral cue, as used in Experiment 3, exhibits a negative influence on the attentional weight. However, it leads to a large effect in the &#964; parameter, which models a delay between the starting times of the encoding processes.
Most of the protocol follows common steps in conducting TOJs and perception experiments in general. Note, however, that the interpretation of the results in terms of TVA is tied to encoding the stimuli into&#160;visual short-term memory. The possibility of performing the TOJ by pure onset detection should be reduced as much as possible. Therefore, as mentioned in protocol step 1.2, it is crucial that the attributes of interest are either automatically encoded (which can be assumed for certain stimuli, e.g., salience pop-outs) or encoding must be facilitated via the task (e.g., reporting the stimulus identity).
It is advisable to inspect the summarized raw data (&#34;probe first&#34; counts across SOAs divided by the number of repetitions) before running the final analysis. This data should follow an S-shaped curve as shown in the psychometric functions in the figures of the Representative Results section. Note that due to the binomially distributed responses, the data points randomly deviate from the ideal path. The deviations increase with a decreasing number of repetitions. With few repetitions, the deviations are often relatively large, obscuring the ideal S shape. However, if the pattern clearly deviates from the usual curve, the mathematical model may need to be adjusted. For instance, when large lateral shifts are observed (as in Experiment 3 of this article),&#160;Alcal&#225;-Quintana and Garc&#237;a-P&#233;rez&#39;s &#964;&#160;parameter can be included. If the curve does not converge to one and zero at its ends, additional lapse parameters21 can be added.
It is possible to perform a formal model comparison as suggested in Alcal&#225;-Quintana and Garc&#237;a-P&#233;rez21 to decide between different models. Using models different from the one outlined in the Introduction, however, may affect whether results can be interpreted in terms of TVA.
In the results reported in this article, we stated the central tendencies of estimated differences along with their 95% HDIs. However, in the Bayesian framework, it is possible to accept or reject that there is no difference between two estimates. For this purpose, a&#160;ROPE (region of practical relevance) must be specified11,13. The ROPE&#160;indicates a small range around zero. Values within this range are considered practically equal to zero. If the ROPE does not overlap with the HDI, the null hypothesis is rejected. Meaningful ROPE limits depend on the research question or application. In contrast to TOJ analysis with traditional means, the TVA-based approach can guide the establishment of meaningful ROPE limits: Due to their meaningful units, the parameters can be related to estimates from other TVA paradigms (e.g., whole reports, see reference3). Furthermore, processing rates can be converted into encoding durations (the expected value of the encoding duration Ex of stimulus x is 1/vx, see reference7) to inform ROPE limits. For example, if researchers are interested in whether an attention manipulation contributes to a reaction time reduction for a participant in a driving simulation, they could reason as follows: Reaction times (including motor components) are in the range of a few hundred milliseconds, therefore, if the attention manipulation changes the overall reaction only a few milliseconds,&#160;the change would be practically zero. Hence a ROPE from -2 to +2 ms could be applied to the difference of reference and probe encoding duration (Er-Ep). If the ROPE of this difference completely includes the HDI, the result that there is no difference can be accepted. If HDI and ROPE do not overlap, the null hypothesis can be rejected. If neither is the case, no such point decision can be made. Further details concerning the Bayesian evaluation approach in general can be found, for instance, in Kruschke&#39;s book13.
Turning to more general issues, for the success of this protocol, it is crucial that there are only two stimuli that generate temporal signals at the target location. For example, a peripheral cue (as in Experiment 3) or masks7 lead to large lateral shifts that cannot be accounted for by the current TVA-based model. Such situations are not uncommon and they were modeled by incorporating&#160;a parameter suggested by Alcal&#225;-Quintana and Garc&#237;a-P&#233;rez21. In this extended model, the &#964; component cannot be clearly linked to a TVA mechanism. There is a tentative connection between &#964; and TVA, but there are some unsolved problems. Indeed, TVA assumes a short delay before encoding starts. Parameter t0, which was discussed in the Introduction, is the maximum ineffective exposure duration before which nothing is encoded at all. The difference t0r-t0p could be understood as &#964;. However, t0 is typically small, around 10 to 20 msec. Furthermore, the theory does not assume that it is influenced by attention. Nevertheless, t0 reductions have been observed in letter recognition7,24. If one accepts this possibility, a further commitment must be made. Parameter &#964; was measured around 50 ms. Given the fact that t0p of the cued stimulus can maximally be decreased by 10 to 20 ms because it is not larger in the first place, most of &#964; would come from increasing t0r of the uncued to 50 to 60 msec. This magnitude is way beyond what sometimes is observed (around 10 msec). As a consequence of &#964;&#8217;s unclear relation to TVA, some important questions cannot be answered. For example, it cannot be decided whether the delays of attended stimuli are reduced or if those of the unattended stimuli are prolonged (resulting in the observed &#964; difference).
The limitations of the technique mentioned above arise from the fact that only two stimuli are explicitly modeled with TVA. To improve on this, future research aims at extending the TVA-based model to more than two stimuli. In particular, explicitly modeling the cue in cued TOJ with TVA is an important objective of subsequent research25.
Advantages of the protocol are the simplicity of the TOJ task which can utilize almost arbitrary stimuli, the thorough theoretical underpinning by TVA, and the Bayesian evaluation scheme. The TVA-based model is a large step forward from traditional model-free approaches. In the past, mostly generic psychometric functions have been fitted to TOJ data. Changes in their summary parameters PSS (point of subjective simultaneity) and DL (difference limen; a measure of discrimination performance) have been linked to attentional manipulations. At times, these parameters are over-interpreted. For example, it is frequently claimed that attention accelerates processing of the attended stimulus, whereas it could also be the case that the unattended stimulus is slowed7. In addition to this weakness, these parameters are rather indirect. They describe the performance in the task and do not characterize the processes that produce it. The model-based analysis of TOJs improves on these drawbacks by providing meaningful parameters based on TVA.

How attention is distributed in space and time is one of the most important factors in human visual perception. Objects that capture attention because of their conspicuity or importance are typically processed faster and with higher accuracy. In behavioral research, such performance benefits have been demonstrated in a variety of experimental paradigms. For instance, allocating attention to the target location speeds up the reaction in probe detection tasks1. Similarly, the accuracy of reporting letters is improved by attention2. Such findings prove that attention enhances processing, but they remain hopelessly mute about how this enhancement is established.
The present paper shows that low-level mechanisms behind attentional advantages can be assessed by measuring the processing speed of individual stimuli in a model-based framework that relates the measurements to fine-grained components of attention. With such a model, the overall processing capacity and its distribution among the stimuli can be inferred from processing speed measurements.
Bundesen&#39;s Theory of Visual Attention (TVA)3 provides a suitable model for this endeavor. It is typically applied to data from letter report tasks. In the following, the fundamentals of TVA are explained and it is shown how they can be extended to model temporal-order judgment (TOJ) data obtained with (almost) arbitrary stimuli. This novel method provides estimates of processing speed and resource distribution which can be readily interpreted. The protocol in this article explains how to plan and conduct such experiments and details how the data can be analyzed.
As mentioned above, the usual paradigm in TVA-based modeling and estimation of attention parameters is the letter report task. Participants report the identities of a set of letters which is briefly flashed and typically masked after a varying delay. Among other parameters, the rate at which visual elements are encoded into visual short-term memory can be estimated. The method has been successfully applied to questions in fundamental and clinical research. For instance, Bublak and colleagues4 assessed which attentional parameters are affected in different stages of age-related cognitive deficits. In fundamental attention research, Petersen, Kyllingsb&#230;k, and Bundesen5 used TVA to model the attentional dwell time effect, the observer&#39;s difficulty in perceiving the second of two targets at certain time intervals. A major drawback of the letter report paradigm is that it requires sufficiently overlearned and maskable stimuli. This requirement limits the method to letters and digits. Other stimuli would require heavy training of participants.
The TOJ paradigm requires neither specific stimuli nor masking. It can be used with any kind of stimuli for which the order of appearance can be judged. This extends the stimulus range to pretty much everything that could be of interest, including direct cross-modal comparisons6.
Investigating attention with TOJs is based on the phenomenon of attentional prior entry which is a measure of how much earlier an attended stimulus is perceived compared to an unattended one. Unfortunately, the usual method for analyzing TOJ data, fitting observer performance psychometric functions (such as cumulative Gaussian or logistic functions), cannot distinguish whether attention increases the processing rate of the attended stimulus or if it decreases the rate of the unattended stimulus7. This ambiguity is a major problem because the question whether the perception of a stimulus is truly enhanced or if it benefits because of the withdrawal of resources from a competing stimulus is a question of both fundamental and practical relevance. For instance, for the design of human-machine interfaces it is highly relevant to know if increasing the prominence of one element works at the expense of another one.
The TOJ task usually proceeds as follows: A fixation mark is presented for a brief delay, typically a randomly drawn interval shorter than a second. Then, the first target is presented, followed after a variable stimulus onset asynchrony (SOA) by the second target. At negative SOAs, the probe, the attended stimulus, is shown first. At positive SOAs, the reference, the unattended stimulus, leads. At an SOA of zero, both targets are shown simultaneously.
Typically, presenting the target refers to switching the stimulus on. Under certain conditions, however, other temporal events, such as a flicker of an already present target or offsets are used8.
In TOJs, responses are collected in an unspeeded manner, usually by keys mapped to the stimulus identities and presentation orders (e.g., if stimuli are squares and diamonds, one key indicates &#34;square first&#34; and another one &#34;diamond first&#34;). Importantly, for the evaluation, these judgments must be converted to &#34;probe first&#34; (or &#34;reference first&#34;) judgments.
In the present work, a combination of the processing model of TVA and the TOJ experimental paradigm is used to eliminate the problems in either individual domain. With this method, readily interpretable speed parameters can be estimated for almost arbitrary visual stimuli, enabling to infer how the observer&#39;s attention is allocated to competing visual elements.
The model is based on TVA&#39;s equations for the processing of individual stimuli, which will be shortly explained in the following. The probability that one stimulus is encoded into visual short-term memory before the other is interpreted as the probability of judging this stimulus as appearing first. The individual encoding durations are exponentially distributed9:
<img alt="Equation 1" src="/files/ftp_upload/54856/54856eq1.jpg" /> (1)
The maximum ineffective exposure duration t0 is a threshold before which nothing is encoded at all. According to TVA, the rate vx,i at which object x is encoded as member of a perceptual category i (such as color or a shape) is given by the rate equation,
<img alt="Equation 2" src="/files/ftp_upload/54856/54856eq2.jpg" />&#160;. (2)
The strength of the sensory evidence that x belongs to category i is expressed in &#951;x,i, and &#946;i is a decision bias for categorizing stimuli as members of category i. This is multiplied by attentional weights. Individual attentional weights wx are divided by the attentional weights of all objects in the visual field. Hence, the relative attentional weight is calculated as
<img alt="Equation 3" src="/files/ftp_upload/54856/54856eq3.jpg" />&#160; (3)
where R represents all categories and &#951;x,i represents the sensory evidence that object x belongs to category j. The value &#960;j is called pertinence of category j and reflects a bias to make categorizations in j. The overall processing capacity C is the sum of all processing rates for all stimuli and categorizations. For a more detailed description of TVA, refer to Bundesen and Habekost&#39;s book9.
In our novel method, Equation 1, which describes the encoding of individual stimuli, is transformed into a model of TOJs. Assuming that selection biases and report categories are constant within an experimental task, the processing rates vp and vr of the two target stimuli probe (p) and reference (r) depend on C and the attentional weights in the form vp= C &#183; wp and vr = C &#183; wr, respectively. The new TOJ model expresses the success probability Pp1st that a participant judges the probe stimulus to be first as a function of the SOA and the processing rates. It can be formalized as follows:
<img alt="Equation 4" src="/files/ftp_upload/54856/54856eq4.jpg" /> (4)
A more detailed description of how this equation is derived from the basic TVA equations is described by T&#252;nnermann, Petersen, and Scharlau7.
For the sake of simplicity, the parameter t0 is omitted in the model in Equation 1. According to the original TVA, t0 should be identical for both targets in the TOJ task, and, therefore, it cancels out. However, this assumption may sometimes be violated (see section Discussion).
For fitting this equation to TOJ data, a hierarchical Bayesian estimation scheme11 is suggested. This approach allows to estimate the attentional weights wp and wr of the probe and reference stimuli and the overall processing rate C. These parameters, the resulting uptake rates vp and vr, and attention-induced differences between them, can be assessed on the subject and group levels along with estimated uncertainties. The hierarchical model is illustrated in Figure 1. During the planning stage for an experiment, convenient Bayesian power analysis can be conducted.
The following protocol describes how to plan, execute and analyze TOJ experiments from which processing speed parameters and attentional weights for visual stimuli can be obtained. The protocol assumes that the researcher is interested in how an attentional manipulation influences the processing speeds of some targets of interest.
<img alt="Figure 1" src="/files/ftp_upload/54856/54856fig1.jpg" />
Figure 1: Graphical model used in the Bayesian estimation procedure. Circles indicate estimated distributions; double circles indicate deterministic nodes. Squares indicate data. The relations are given on the right side of the figure. The nodes outside the rounded frames (&#8220;plates&#8221;) represent mean and dispersion estimates of TVA parameters (see Introduction) on the group level. In the &#8220;j Subjects&#8221; plate, it can be seen how attentional weights (w) are combined with the overall processing rates (C) to from stimulus processing rates (v) on the subject level. Plate &#8220;i SOAs&#8221; shows how these TVA parameters are then transformed (via the function Pp1st described in the Introduction) into the success probability (&#952;) for the binomially distributed responses at each SOA. Therefore, the &#952; together with the repetitions of the SOA (n) describe the data points (y). For more details on notation and interpretation of graphical models, refer to Lee and Wagenmakers23. Note that for the sake of clarity, the nodes that represent differences of parameters have been omitted. These deterministic parameters are indicated in the figures of the experimental results instead. <a href="https://cloudflare-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54856/54856fig1large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

<table><tbody><tr><td>Personal Computer</td><td></td><td></td><td></td></tr><tr><td>(Open Source) Experimentation and evaluation software</td><td></td><td></td><td></td></tr></tbody></table>

measuring attention and visual processing speed by model-based analysis of temporal-order judgments

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The proposed method is based on a new and synergistic combination of three components: the temporal-order judgments (TOJ) paradigm, Bundesen&#39;s Theory of Visual Attention (TVA), and a hierarchical Bayesian estimation framework. The method provides readily interpretable parameters, which are supported by the theoretical and neurophysiological underpinnings of TVA. Using TOJs, TVA-based estimates can be obtained for a broad range of stimuli, whereas traditional paradigms used with TVA are mainly limited to letters and digits. Finally, the meaningful parameters of the proposed model allow for the establishment of a hierarchical Bayesian model. Such a statistical model allows assessing results in one coherent analysis both on the subject and the group level.
To demonstrate the feasibility and versatility of this new approach, three experiments are reported with attention manipulations in synthetic pop-out displays, natural images, and a cued letter-report paradigm.

Watch this Scientific Journal Video about Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments at JoVE.com

Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments

Temporal-order judgments can be used to estimate processing speed parameters and attentional weights and thereby to infer the mechanisms of attentional processing. This methodology can be applied to a wide range of visual stimuli and works with many attention manipulations.

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The ...

measuring-attention-visual-processing-speed-model-based-analysis

Nanyang Technological University

Research

JoVE Journal

Behavior

9.8K Views.  Paderborn University. Temporal-order judgments can be used to estimate processing speed parameters and attentional weights and thereby to infer the mechanisms of attentional processing. This methodology can be applied to a wide range of visual stimuli and works with many attention manipulations.

Video: Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments

Automated Visual Cognitive Tasks for Recording Neural Activity Using a Floor Projection Maze

Neuropsychological tasks used in primates to investigate mechanisms of learning and memory are typically visually guided cognitive tasks. We have developed visual cognitive tasks for rats using the Floor Projection Maze1,2&#160;that are optimized for visual abilities of rats permitting stronger comparisons of experimental findings with other species.
In order to investigate neural correlates of learning and memory, we have integrated electrophysiological recordings into fully automated cognitive tasks on the Floor Projection Maze1,2. Behavioral software interfaced with an animal tracking system allows monitoring of the animal's behavior with precise control of image presentation and reward contingencies for better trained animals. Integration with an in vivo electrophysiological recording system enables examination of behavioral correlates of neural activity at selected epochs of a given cognitive task.
We describe protocols for a model system that combines automated visual presentation of information to rodents and intracranial reward with electrophysiological approaches. Our model system offers a sophisticated set of tools as a framework for other cognitive tasks to better isolate and identify specific mechanisms contributing to particular cognitive processes.

We describe protocols for training rats for chronic electrophysiological recordings in fully automated cognitive tasks on a Floor Projection Maze.

Neuropsychological tasks used in primates to investigate mechanisms of learning and memory are typically visually guided cognitive tasks. We have ...

A Dual Task Procedure Combined with Rapid Serial Visual Presentation to Test Attentional Blink for Nontargets

When viewers search for targets in a rapid serial visual presentation (RSVP) stream, if two targets are presented within about 500 msec of each other, the first target may be easy to spot but the second is likely to be missed. This phenomenon of attentional blink (AB) has been widely studied to probe the temporal capacity of attention for detecting visual targets. However, with the typical procedure of AB experiments, it is not possible to examine how the processing of non-target items in RSVP may be affected by attention. This paper describes a novel dual task procedure combined with RSVP to test effects of AB for nontargets at varied stimulus onset asynchronies (SOAs). In an exemplar experiment, a target category was first displayed, followed by a sequence of 8 nouns. If one of the nouns belonged to the target category, participants would respond &#8216;yes&#8217; at the end of the sequence, otherwise participants would respond &#8216;no&#8217;. Two 2-alternative forced choice memory tasks followed the response to determine if participants remembered the words immediately before or after the target, as well as a random word from another part of the sequence. In a second exemplar experiment, the same design was used, except that 1) the memory task was counterbalanced into two groups with SOAs of either 120 or 240 msec and 2) three memory tasks followed the sequence and tested remembrance for nontarget nouns in the sequence that could be anywhere from 3 items prior the target noun position to 3 items following the target noun position. Representative results from a previously published study demonstrate that our procedure can be used to examine divergent effects of attention that not only enhance targets but also suppress nontargets. Here we show results from a representative participant that replicated the previous finding.&#160;

This paper describes a novel dual task procedure combined with Rapid Serial Visual Presentation to look at attentional blink at varied stimulus onset asynchronies. This experimental procedure differed from others in that it tested attentional blink for nontargets.

When viewers search for targets in a rapid serial visual presentation (RSVP) stream, if two targets are presented within about 500 msec of each other, the ...

Methods to Test Visual Attention Online

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative data samples than can typically be collected on college campuses. However, before such methods can be widely adopted, a number of technological challenges must be overcome &#8211; in particular in experiments where tight control over stimulus properties is necessary. Here we present methods for collecting performance data on two tests of visual attention. Both tests require control over the visual angle of the stimuli (which in turn requires knowledge of the viewing distance, monitor size, screen resolution, etc.) and the timing of the stimuli (as the tests involve either briefly flashed stimuli or stimuli that move at specific rates). Data collected on these tests from over 1,700 online participants were consistent with data collected in laboratory-based versions of the exact same tests. These results suggest that with proper care, timing/stimulus size dependent tasks can be deployed in web-based settings.

To replicate laboratory settings, online data collection methods for visual tasks require tight control over stimulus presentation. We outline methods for the use of a web application to collect performance data on two tests of visual attention.

Online data collection methods have particular appeal to behavioral scientists because they offer the promise of much larger and much more representative ...

Testing Sensory and Multisensory Function in Children with Autism Spectrum Disorder

In addition to impairments in social communication and the presence of restricted interests and repetitive behaviors, deficits in sensory processing are now recognized as a core symptom in autism spectrum disorder (ASD). Our ability to perceive and interact with the external world is rooted in sensory processing. For example, listening to a conversation entails processing the auditory cues coming from the speaker (speech content, prosody, syntax) as well as the associated visual information (facial expressions, gestures). Collectively, the &#8220;integration&#8221; of these multisensory (i.e., combined audiovisual) pieces of information results in better comprehension. Such multisensory integration has been shown to be strongly dependent upon the temporal relationship of the paired stimuli. Thus, stimuli that occur in close temporal proximity are highly likely to result in behavioral and perceptual benefits &#8211; gains believed to be reflective of the perceptual system&#39;s judgment of the likelihood that these two stimuli came from the same source. Changes in this temporal integration are expected to strongly alter perceptual processes, and are likely to diminish the ability to accurately perceive and interact with our world. Here, a battery of tasks designed to characterize various aspects of sensory and multisensory temporal processing in children with ASD is described. In addition to its utility in autism, this battery has great potential for characterizing changes in sensory function in other clinical populations, as well as being used to examine changes in these processes across the lifespan.

We describe how to implement a battery of behavioral tasks to examine the processing and integration of sensory stimuli in children with ASD. The goal is to characterize individual differences in temporal processing of simple auditory and visual stimuli and relate these to higher order perceptual skills like speech perception.

In addition to impairments in social communication and the presence of restricted interests and repetitive behaviors, deficits in sensory processing are ...

Measurement of Neurophysiological Signals of Ignoring and Attending Processes in Attention Control

Attention control is the ability to selectively attend to some sensory signals while ignoring others. This ability is thought to involve two processes: enhancement of sensory signals that are to be attended and the attenuation of sensory signals that are to be ignored. The overall strength of attentional modulation is often measured by comparing the amplitude of a sensory neural response to an external input when attended versus when ignored. This method is robust for detecting attentional modulation, but precludes the ability to assess the separate dynamics of attending and ignoring processes. Here, we describe methodology to measure independently the neurophysiological signals of attending and ignoring using the intermodal attention task (IMAT). This task, when combined with electroencephalography, isolates neurophysiological sensory responses in auditory and visual modalities, when either attending or ignoring, with respect to a passive control. As a result, independent dynamics of attending and of a ignoring can be assessed in either modality. Our results using this task indicate that the timing and cortical sources of attending and ignoring effects differ, as do their contributions to the attention modulation effect, pointing to unique neural trajectories and demonstrating sample utility of measuring them separately.

Attention control comprises enhancement of target signals and attenuation of distractor signals. We describe an approach to measure separately but concurrently, the neurophysiology of attending and ignoring in sustained intermodal attention, utilizing a passive control condition during which neither process is continuously engaged.

Attention control is the ability to selectively attend to some sensory signals while ignoring others. This ability is thought to involve two processes: ...

Using Rapid Serial Visual Presentation to Measure Set-Specific Capture, a Consequence of Distraction While Multitasking

This method uses a rapid serial visual presentation (RSVP) paradigm to measure the cost of distraction when participants maintain multiple search goals. The protocol identifies two types of distraction within a single task - contingent attentional capture and set-specific capture - that represent different types of limitations of cognitive processing. Participants search for letters in two or more &#34;target&#34; ink colors (e.g., green and orange) within a continuous RSVP stream of heterogeneously colored letters, while ignoring two peripheral RSVPs of letters. Upon detecting a target, participants are to identify the letter. On some trials, target-colored distractors appear in the periphery just prior to the presentation of a target, causing a drop in target identification performance. Contingent attentional capture is observed by examining performance on trials in which the peripheral distractor is the same color as the target on that trial (e.g., both orange). Set-specific capture is represented by performance on trials in which the peripheral distractor is target-colored (e.g., orange), but not the same color as the target on that trial (e.g., green.) By varying the amount of time (i.e., the number of stimuli appearing) between the presentation of the distractor and the target, researchers can observe how participants recover from these distraction costs over time. As compared to static displays that are often used to measure contingent attentional capture, the dynamic display produces much larger effects, allowing the researcher to identify subtle effects of smaller manipulations. An unusual aspect of our design is that it employs a continuous display; &#34;filler&#34; stimuli connect one trial to the next seamlessly, and participants respond during this interval whenever they detect a target. The continuous display reduces chance performance to near-zero levels (rather than 50%) and provides researchers with a more sensitive measure of performance differences across trial types.

This method uses a dynamic visual display to index costs of distraction during visual search, including both &#34;contingent attentional capture&#34; and &#34;set-specific capture,&#34; which is a cost of distraction that occurs when the participants maintain multiple search goals simultaneously. This method has revealed basic mechanisms and limitations of visual attention.

This method uses a rapid serial visual presentation (RSVP) paradigm to measure the cost of distraction when participants maintain multiple search goals. ...

Investigating the Deployment of Visual Attention Before Accurate and Averaging Saccades via Eye Tracking and Assessment of Visual Sensitivity

This experimental protocol was designed to investigate whether visual attention is obligatorily deployed at the endpoint of saccades. To this end, we recorded the eye position of human participants engaged in a saccade task via eye tracking and assessed visual orientation discrimination performance at various locations during saccade preparation. Importantly, instead of using a single saccade target paradigm for which the saccade endpoint typically coincides roughly with the target, this protocol comprised the presentation of two nearby saccade targets, leading to a distinct spatial dissociation between target locations and saccade endpoint on a substantial number of trials. The paradigm allowed us to compare presaccadic visual discrimination performance at the endpoint of accurate saccades (landing at one of the saccade targets) and of averaging saccades (landing at an intermediate location in between the two targets). We observed a selective enhancement of visual sensitivity at the endpoint of accurate saccades but not at the endpoint of averaging saccades. Rather, before the execution of averaging saccades, visual sensitivity was equally enhanced at both targets, suggesting that saccade averaging follows from unresolved attentional selection among the saccade targets. These results argue against a mandatory coupling between visual attention and saccade programming based on a direct measure of presaccadic visual sensitivity rather than saccadic reaction times, which have been used in other protocols to draw similar conclusions. While our protocol provides a useful framework to investigate the relationship between visual attention and saccadic eye movements at the behavioral level, it can also be combined with electrophysiological measures to extend insights at the neuronal level.

This experimental protocol combined eye tracking and the assessment of presaccadic visual sensitivity in a dual task paradigm, consisting of a free choice saccade task and a visual discrimination task, to investigate the deployment of visual spatial attention before both, accurate and averaging saccades.

This experimental protocol was designed to investigate whether visual attention is obligatorily deployed at the endpoint of saccades. To this end, we ...

Investigating Oculomotor Responses in Visual Duration Perception Tasks

Eye Movements in Visual Duration Perception: Disentangling Stimulus from Time in Predecisional Processes

Eye-tracking methods may allow the online monitoring of cognitive processing during visual duration perception tasks, where participants are asked to estimate, discriminate, or compare time intervals defined by visual events like flashing circles. However, and to our knowledge, attempts to validate this possibility have remained inconclusive so far, and results remain focused on offline behavioral decisions made after stimulus appearance. This paper presents an eye-tracking protocol for exploring the cognitive processes preceding behavioral responses in an interval comparison task, where participants viewed two consecutive intervals and had to decide whether it speeded up (first interval longer than second) or slowed down (second interval longer). 
Our main concern was disentangling oculomotor responses to the visual stimulus itself from correlates of duration related to judgments. To achieve this, we defined three consecutive time windows based on critical events: baseline onset, the onset of the first interval, the onset of the second interval, and the end of the stimulus. We then extracted traditional oculomotor measures for each (number of fixations, pupil size) and focused on time-window-related changes to separate the responses to the visual stimulus from those related to interval comparison per se. As we show in the illustrative results, eye-tracking data showed significant differences that were consistent with behavioral results, raising hypotheses on the mechanisms engaged. This protocol is embryonic and will require many improvements, but it represents an important step forward in the current state of art.

Author Spotlight: Exploring the Link Between Time Perception of Visual Stimuli and Reading Skills

We present a protocol that employs eye tracking to monitor eye movements during an interval comparison (duration perception) task based on visual events. The aim is to provide a preliminary guide to separate oculomotor responses to duration perception tasks (comparison or discrimination of time intervals) from responses to the stimulus itself.

Eye-tracking methods may allow the online monitoring of cognitive processing during visual duration perception tasks, where participants are asked to ...

Spatial Cueing

Source: Laboratory of Jonathan Flombaum&#8212;Johns Hopkins University
Attention refers to the limited human ability to select some information for processing at the expense of other stimuli in the environment. Attention operates in all sensory modalities: vision, hearing, touch, even taste and smell. It is most often studied in the visual domain though. A common way to study visual attention is with a spatial cueing paradigm. This paradigm allows researchers to measure the consequences of focusing visual attention in some locations and not others. This paradigm was developed by psychologist Michael Posner in the late 70s and early 80s in a series of papers in which he likened attention to a spotlight, selectively illuminating some portion of a scene.1,2 This video demonstrates standard procedures for a spatial cueing experiment to investigate visual attention.

Spatial Cueing Task to Investigate Covert Visual Attention

Source: Laboratory of Jonathan Flombaum&#8212;Johns Hopkins University
Attention refers to the limited human ability to select some information for ...

Education

JoVE Science Education

Psychology

Sensation and Perception

The Attentional Blink

Source: Laboratory of Jonathan Flombaum&#8212;Johns Hopkins University
In order for recognition of a certain stimulus to take place, visual attention needs to be directed towards said stimulus. To the earliest parts of the visual system, objects are not objects, they are collections of visual features-lines, corners, changes in texture, color, and light. Attention is the resource that is necessary for later processing in order to recognize what a given bundle of features adds up to. This makes attention a central focus of research. One especially important set of questions concerns how people sustain attention, that is, the extent to which they can continuously maintain a focus of attention from moment-to-moment. It is now known that sustained attention takes great effort. When attention needs to be focused very rapidly on something that is moving or changing very quickly, the effort involved causes a momentary lapse in attention once it is disengaged. This kind of lapse in attention is called an attentional blink. It is like the brain blinks for a moment, shutting down attention for a rest. Stimuli that appear during an attentional blink will not be perceived. 
In 1992, a group of researchers devised a paradigm to study the attentional blink, and the paradigm has come to be known by the same name.1 It demonstrates some of the challenges to maintaining focused attention. This video demonstrates how to implement the attentional blink paradigm in order to study sustained visual attention.

Sustained Visual Attention and the Attentional Blink Paradigm

Source: Laboratory of Jonathan Flombaum&#8212;Johns Hopkins University
In order for recognition of a certain stimulus to take place, visual attention ...