This work is supported by the Spanish State Research Agency and the European Regional Development Fund (ERDF) under the PALLAS (TIN2016-80515-R AEI/EFRD, EU) project.

All methods described below have been approved by the regional government of Galicia&#39;s committee for research ethics of Pontevedra-Vigo-Ourense (reg. code 2017/336). The protocol was implemented for first year students at the School of Telecommunication Engineering - University of Vigo, both in a comfortable laboratory room and in several lectures and practice sessions of a bachelor&#39;s degree course on Computer Architectures.
1. Prepare the Devices
<ol>
	<li>Connect the smartphone and tablet device to a stable internet connection.</li>
	<li>Turn on Bluetooth communications in the smartphone.</li>
	<li>In the smartphone, search in the corresponding official app store the wrist wearable application. Download and install it.</li>
	<li>In the smartphone, search for the PhysiologicalSignals app to capture physiological signals. Download and install it. 
	NOTE: Currently, the app is a beta version and access can be provided by request.</li>
	<li>In the tablet, search for the StressTest app to be used in the research laboratory experiments. Download and install it. 
	NOTE: Currently, the app is a beta version and access can be provided by request.</li>
	<li>Turn on the COTS wrist wearable device and place the wearable.</li>
	<li>In the smartphone, open the official COTS wrist wearable application. 
	NOTE: The app will proceed to synchronize the wearable device with the smartphone. In some devices, an e-mail address is required.</li>
	<li>In the smartphone, open the PhysiologicalSignals app.
	<ol>
		<li>In case of being notified of a sensor access request, accept it.</li>
		<li>Check the device. Wait for the PhysiologicalSignals app to display the word Weared in green. 
		NOTE: This indicate that the wearable device has been detected and therefore the transmission of information from the sensors to the smartphone is enabled. If this message does not appear repeat from step 1.6.</li>
	</ol>
	</li>
</ol>
2. The Laboratory Phase
<ol>
	<li>Prepare the laboratory setting. Choose a comfortable and non-disturbing room without distracting noise and with a comfortable temperature (between 22 &#176;C and 26 &#176;C).</li>
	<li>Turn on the wrist wearable device, place it around the subject&#180;s non-dominant wrist and place the headphones on the head of the student. Fit the wearable tightly but comfortably around the wrist.</li>
	<li>Connect the smartphone and tablet to a stable internet connection and verify that the Bluetooth connection is active.</li>
	<li>In the smartphone, launch the PhysiologicalSignals app.
	<ol>
		<li>Wait for the app to display the word Weared in green.</li>
		<li>Select the Change User option in the left configuration menu and provide the ID of the subject who will complete the tests and click Save.</li>
	</ol>
	</li>
	<li>In a laptop, access the dashboard and enter the test administrator&#39;s ID and password. 
	NOTE: Currently, for private and security concerns, access to the dashboard is only available under request.
	<ol>
		<li>Select the subject ID and the subject&#39;s stress tab.</li>
		<li>Check the physiological signals evolution and wait for the wearable device to reach thermal stability before starting the experiment. 
		NOTE: The thermal stability is identified as a plateau in the graph.</li>
	</ol>
	</li>
	<li>In the tablet, launch the StressTest application.
	<ol>
		<li>Explain to the subject the four laboratory tasks. Show some of the screens and actions to perform during each one of the tasks. 
		NOTE: This is very important, because the subject should feel stressed or relaxed in accordance to the performed activities, and not fear or concern about what is going to happen.</li>
		<li>Tell the student not to rest their arms on the table and to use the hand where the wearable device is placed to perform the activities.</li>
		<li>Enter the same user ID as in step 2.4.2 and click the arrow.</li>
	</ol>
	</li>
	<li>Launch the video task and give full control to the student.
	<ol>
		<li>Observe that the task is carried out without incident.</li>
		<li>When the task is finished, check that the subject provides the perceived stress.</li>
	</ol>
	</li>
	<li>Launch the Stroop Color task (SCWT) consecutively for levels 1, 2 and 3.
	<ol>
		<li>For each level, observe that the subtask is carried out without incident.</li>
		<li>When each subtask is finished, check that the subject provides the perceived stress.</li>
		<li>Only for level 3 and only in case the subject does not solve it after 4 minutes, terminate the task by pressing the arrow located at the top of the screen.</li>
	</ol>
	</li>
	<li>Launch the Paced Auditory Serial Addition test (PASAT).
	<ol>
		<li>Observe that the task is carried out without incident.</li>
		<li>In case the subject does not solve the PASAT test after 4 minutes, terminate the task by pressing the arrow located at the top of the screen.</li>
		<li>When the task is finished, check that the subject provides the perceived stress.</li>
	</ol>
	</li>
	<li>Launch the Hyperventilation test.
	<ol>
		<li>Observe the evolution of HR using the dashboard. If physiological signals do not change significantly, ask the subject to increase inspiration and expiration rates gradually.</li>
		<li>In case the subject feels dizziness or uncomfortable halt this task. In any case, complete the task after four minutes.</li>
		<li>When the task is finished, check that the subject provides the perceived stress.</li>
	</ol>
	</li>
</ol>
3. The Classroom Phase
<ol>
	<li>Turn on the wrist wearable device and place the wearable around the subject&#180;s non-dominant wrist. Fit the wearable tightly but comfortably around the wrist.</li>
	<li>Connect the smartphone to a stable internet connection and verify the Bluetooth connection is active.</li>
	<li>In the smartphone, launch the PhysiologicalSignals app.
	<ol>
		<li>Wait for the app to display the word Weared in green.</li>
		<li>Select in the configuration menu the Change User option, provide the ID of the subject who will complete the tests and click Save.</li>
	</ol>
	</li>
	<li>In a laptop, access the dashboard and enter the test administrator&#39;s ID and password.
	<ol>
		<li>Select the subject ID and the subject&#39;s stress tab.</li>
		<li>Check the evolution of physiological signals.</li>
	</ol>
	</li>
	<li>Take annotations about any relevant event occurring in the classroom in relation to the student-teacher interaction. 
	NOTE: Relevant information and basic events will be used to label physiological samples afterwards. Example events are a question from the teacher to the student, or a theoretical explanation is initiated.</li>
	<li>At the end of the lecture, ask the subject to complete the questionnaire about their level of stress at specific times during the session, according to a 5-level scale.</li>
</ol>
4. Data Analysis
<ol>
	<li>In a laptop, access the dashboard and enter the test administrator&#39;s ID and password.
	<ol>
		<li>Select the subject ID and the subject&#39;s stress tab.</li>
		<li>Select the day of a classroom experiment.</li>
	</ol>
	</li>
	<li>Label the samples of the subject by identifying activities and perceived stress levels.
	<ol>
		<li>Identify lecture-room activities and their duration according to the starting and finishing times and their types.</li>
		<li>For each activity, select a perceived stress level.</li>
	</ol>
	</li>
	<li>For each subject and each session, download the file with the tagged samples. 
	NOTE: A comma-separated-values (CSV) file is created for each student, each row reflecting the values of the physiological signals with their standard deviation, slope and diff, the activity type, the activity-based stress (i.e., the stress associated by default to the activity) and the subject perceived stress.</li>
	<li>Launch the data analytics package.
	<ol>
		<li>Choose a set of classifiers (e.g., SVM, C4.5, k-NN, Random Forest, Na&#239;ve Bayes and Zero R) and import the CSV file for all students for each session.</li>
		<li>Train and evaluate classifiers using the 10-fold cross-validation technique. 
		NOTE: Depending on the analyses, activity type, activity-based stress or stress perceived, shall be selected as dependent variable for the analysis.</li>
		<li>Finally, check the results for accuracy and error rates.</li>
	</ol>
	</li>
</ol>

<ol>
	<li>IDC. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Forecasts Wearables Shipments to Reach 213.6 Million Units Worldwide in 2020 with Watches and Wristbands Driving Volume While Clothing and Eyewear Gain Traction. , (2016).</li><li>Piwek, L., Ellis, D. A., Andrews, S., Joinson, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+rise+of+consumer+health+wearables:promises+and+barriers.">The rise of consumer health wearables:promises and barriers.</a> PLoS Medicine. 13 (2), (2016).</li><li>Rudner, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Interrogation+of+Patient+Smartphone+Activity+Tracker+to+Assist+Arrhythmia+Management.">Interrogation of Patient Smartphone Activity Tracker to Assist Arrhythmia Management.</a> Annals of Emergency Medicine. 68 (3), 292-294 (2016).</li><li>Gao, Y., Li, H., Luo, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=An+empirical+study+of+wearable+technology+acceptance+in+healthcare.">An empirical study of wearable technology acceptance in healthcare.</a> Industrial Management &amp; Data Systems. 115 (9), 1704-1723 (2015).</li><li>Lukowicz, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Glass-physics:+using+google+glass+to+support+high+school+physics+experiments.">Glass-physics: using google glass to support high school physics experiments.</a> Proceedings of the 2015 ACM International Symposium on Wearable Computers - ISWC '15. , 151-154 (2015).</li><li>Sapargaliyev, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Wearables in Education: Expectations and Disappointments. , 73-78 (2015).</li><li>de Arriba P&#233;rez, F., CaeiroRodr&#237;guez, M., Santos Gago, J. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=How+do+you+sleep?+Using+off+the+shelf+wrist+wearables+to+estimate+sleep+quality,+sleepiness+level,+chronotype+and+sleep+regularity+indicators.">How do you sleep? Using off the shelf wrist wearables to estimate sleep quality, sleepiness level, chronotype and sleep regularity indicators.</a> Journal of Ambient Intelligence and Humanized Computing. , 1-21 (2017).</li><li>Espinosa, H. G., Lee, J., Keogh, J., Grigg, J., James, D. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=On+the+Use+of+Inertial+Sensors+in+Educational+Engagement+Activities.">On the Use of Inertial Sensors in Educational Engagement Activities.</a> Procedia Engineering. 112, 262-266 (2015).</li><li>Travers, C. J., Cooper, C. L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> El Estre&#769;s de los profesores la presio&#769;n en la actividad docente. , (1997).</li><li>Maslach, C., Jackson, S. E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+measurement+of+experienced+burnout.">The measurement of experienced burnout.</a> Journal of occupational Behavior. 2, 99-113 (1981).</li><li>Maslach, C., Jackson, S., Leiter, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Maslach Burnout Inventory. , (1986).</li><li>Kitsantas, A., Winsler, A., Huie, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Self-Regulation+and+Ability+Predictors+of+Academic+Success+During+College:+A+Predictive+Validity+Study.">Self-Regulation and Ability Predictors of Academic Success During College: A Predictive Validity Study.</a> Journal of Advanced Academics. 20, (2008).</li><li>Deberard, C., Scott, M., Glen, I., Spielmans, D. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Julka+Predictors+of+academic+achievement+and+retention+among+college+freshmen:+a+longitudinal+study.">Julka Predictors of academic achievement and retention among college freshmen: a longitudinal study.</a> College Student Journal. 381, 66-80 (2004).</li><li>Healey, J. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Wearable and automotive systems for affect recognition from physiology. , (2000).</li><li>Vrijkotte, T. G. M., van Doornen, L. J. P., de Geus, E. J. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Effects+of+Work+Stress+on+Ambulatory+Blood+Pressure,+Heart+Rate,+and+Heart+Rate+Variability.">Effects of Work Stress on Ambulatory Blood Pressure, Heart Rate, and Heart Rate Variability.</a> Hypertension. 35 (4), (2000).</li><li>Karthikeyan, P., Murugappan, M., Yaacob, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Descriptive+Analysis+of+Skin+Temperature+Variability+of+Sympathetic+Nervous+System+Activity+in+Stress.">Descriptive Analysis of Skin Temperature Variability of Sympathetic Nervous System Activity in Stress.</a> Journal of Physical Therapy Science. 24 (12), 1341-1344 (2012).</li><li>de Santos Sierra, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Design, implementation and evaluation of an unconstrained and contactless biometric system based on hand geometry and stress detection. , (2012).</li><li>Natale, V., Drejak, M., Erbacci, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Monitoring+sleep+with+a+smartphone+accelerometer.">Monitoring sleep with a smartphone accelerometer.</a> Sleep and Biological Rhythms. , (2012).</li><li>Guo, F., Li, Y., Kankanhalli, M., Brown, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=An+evaluation+of+wearable+activity+monitoring+devices.">An evaluation of wearable activity monitoring devices.</a> Proceedings of the 1st ACM international workshop on Personal data meets distributed multimedia. , (2013).</li><li>Wallen, M. P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Accuracy+of+Heart+Rate+Watches:+Implications+for+Weight+Management.">Accuracy of Heart Rate Watches: Implications for Weight Management.</a> PLOS ONE. 11 (5), e0154420 (2016).</li><li>Wang, R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Accuracy+of+Wrist-Worn+Heart+Rate+Monitors.">Accuracy of Wrist-Worn Heart Rate Monitors.</a> JAMA Cardiology. 2 (1), 104 (2017).</li><li>de Arriba P&#233;rez, F., CaeiroRodr&#237;guez, M., Santos Gago, J. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Collection+and+Processing+of+Data+from+Wrist+Wearable+Devices+in+Heterogeneous+and+Multiple-User+Scenarios.">Collection and Processing of Data from Wrist Wearable Devices in Heterogeneous and Multiple-User Scenarios.</a> Sensors. 16 (9), 1538 (2016).</li><li>IDC. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Wearables Aren't Dead, They're Just Shifting Focus as the Market Grows 16.9% in the Fourth Quarter, According to IDC. , (2017).</li><li>Mark, H., Ian, W., Eibe, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Data Mining: Practical Machine Learning Tools and Techniques. , (2011).</li><li>de Arriba P&#233;rez, F., Santos Gago, J. M., Caeiro Rodr&#237;guez, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Analytics+of+biometric+data+from+wearable+devices+to+support+teaching+and+learning+activities.">Analytics of biometric data from wearable devices to support teaching and learning activities.</a> Journal of Information Systems Engineering &amp; Management. 1, 41-54 (2016).</li><li>Machine Learning Group at the University of Waikato. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Weka 3 - Data Mining with Open Source Machine Learning Software in Java. , (2018).</li><li>Zhai, J., Barreto, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Stress+Detection+in+Computer+Users+Based+on+Digital+Signal+Processing+of+Noninvasive+Physiological+Variables.">Stress Detection in Computer Users Based on Digital Signal Processing of Noninvasive Physiological Variables.</a> 2006 International Conference of the IEEE Engineering in Medicine and Biology Society. , 1355-1358 (2006).</li><li>Tombaugh, T. N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+comprehensive+review+of+the+Paced+Auditory+Serial+Addition+Test+(PASAT).">A comprehensive review of the Paced Auditory Serial Addition Test (PASAT).</a> Archives of Clinical Neuropsychology. 21 (1), 53-76 (2006).</li><li>Fan, Q., Wang, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+real-time+realization+of+filtering+of+speech+with+DSP+TMS320VC5416+Chip.">The real-time realization of filtering of speech with DSP TMS320VC5416 Chip.</a> 2010 International Conference on Educational and Information Technology. , (2010).</li><li>Gonz&#225;lez Barajas, J. E., Velandia C&#225;rdenas, C., Nieto Camacho, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Implementaci&#243;n+de+filtro+digital+en+tiempo+real+para+detecci&#243;n+de+la+onda+R.">Implementaci&#243;n de filtro digital en tiempo real para detecci&#243;n de la onda R.</a> Revista Tecno L&#243;gicas. 18 (34), 75-86 (2015).</li><li>Mesirov, J. P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Computer+science.+Accessible+reproducible+research.">Computer science. Accessible reproducible research.</a> Science (New York, N.Y.). 327 (5964), 415-416 (2010).</li><li>American Journal Experts. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> How to Write an Easily Reproducible Protocol. , (2018).</li></ol>

The authors have nothing to disclose.

COTS wearable devices are among the most popular consumer electronics products available today. These devices are typically used to monitor physical activities, but their capabilities and performance could be of great interest in other areas. In this paper, a protocol to evaluate the use of COTS wearable devices for estimating stress in learning environments is discussed. The definition of such a protocol is especially relevant in order to analyze different solutions involving wearables and machine learning algorithms. The protocol is intended to be used in educational settings, where the validation of stress detection procedures and their eventual introduction may provide significate benefits. For example the use of wearable devices can contribute to reduce the high levels of stress associated to the so-called burnout syndrome9,10,11, and as a consequence the dropout rate at universities12,13, while improving academic performance.
A critical aspect to consider is the Bluetooth link between the wearable and the smartphone. This wireless connection between both devices may be broken during the test, so it is necessary to pay special attention to it through the visualization of the data collected in the dashboard. Although recovery is performed automatically after a short period of time (i.e., an interval ranging from 1 to 10 minutes), this interruption may cause the loss of the samples in that interval. To reduce the amount of information lost, it may be convenient to manually reset the smartphone device. Other aspect to be considered is the initial skin temperature sensor value, as it may affect the achievement of skin stability, which may be delayed up to 10 minutes.
The main advantages of the protocol proposed in this research are its applicability to a large group of students, its minimal need for support using automated mobile apps, its simplicity in the preparation of the devices involved in the experiment and its low intrusiveness while carrying out the classroom phase. This protocol provides a fast and simple method applicable in controlled environments, such as classrooms or university laboratories. Besides, technological abilities of participating students are not an issue, as the protocol is based in straightforward technical concepts understandable by an average university student independently of their academic field. As stated in the literature31, reproducibility in experimental sciences requires a thorough and clear description of the protocols applied and the results thereof. The protocol discussed in this paper has been designed in a modular way according to simple, straightforward steps, which facilitates the reproduction of the experiments discussed and their extension32. Among the most relevant design aspects facilitating reproducibility, we can name the conciseness of the laboratory phase and its automated implementation by means of standalone mobile apps. Additionally, the classroom phase does not require any interaction with the students beyond academic activities. Most students pointed out the simplicity of the process, and no complaints were reported in relation to their involvement in the experiments. To sum up, collected evidence so far indicates that this protocol may be applied to subjects with a broader profile and in fields different to education, such as health facilities or the working place. Besides, this protocol offers the possibility to study several machine learning solutions with which to test the best algorithms to implement depending on the requirements of the experiments and on the wearable device selected. The use of applications to induce stress and to provide a dashboard to display and tag samples facilitates the training of custom stress models in a single laboratory session.
The main limitations of the proposed solution are related on the subjects&#39; variability and the reproducibility of academic activities. Recreating exactly the same conditions and situations taking place in lecture sessions is practically impossible. On the other hand, the stress experienced by each student is very personal, as in general there are different responses to the same stimuli. In addition, there are hardware-related issues related to the wearable devices themselves, such as different access methods, different sensors, access to physiological signals in real time, or battery life. These technical requirements restrict eligible wearables to a limited range of devices. In our case, eligible devices include those compatible with smart Bluetooth capabilities and smart bands with a SDK compatible with major SO smartphone devices. The number of compatible devices is expected to increase along the next years.
The proposed protocol is intended to serve as an instrument to eventually define richer student models than those presently used in learning management systems or student information systems. For example, the new information captured with the wearable device according to the protocol discussed could be applied to the early detection of situations affecting performance such as fatigue or stress, and to guide students to overcome these situations. An alternative to this protocol may be based on wearable devices worn also outside the classroom in order to detect variations in physiological signals over a longer period of time. This approach involves several challenges, such as a constantly changing ambient temperature, or the subject under study being forced to always be close to their smartphone to prevent data loss. Finally, this protocol may be also applied to other courses and educational levels, which would facilitate the capture of additional evidence on how stress influences academic performance for students with different skills or fields of study.

State-of-the-art wearable technologies are widely available, and their application environments are continuously expanding. We can find in the market many different devices, among which COTS wrist wearables1, such as smart watches and smart bands, are popular among athletes as a personal physical fitness monitoring tool2. By applying data analytic techniques, the data obtained using these devices can be processed to provide indicators such as general physical state, sleep quality or recovery factor. The demonstrated applicability in this area raised interest in the academic community about their possible application to other fields, especially in the health domain3,4, although the strict requirements of clinical trials limit their introduction. However, in a less demanding context such as education, we can find in the literature recent investigations involving the use of different types of wearable devices, both related to teaching activities5,6 and to the estimation of certain characteristics of the student such as sleep patterns7, or the analysis of students&#39; engagement in different educational activities8.
In our case, we focus on analyzing COTS wrist wearable devices as means to collect physiological signals that would eventually facilitate stress estimation, which in turn is a key aspect in educational contexts. Stress has a relevant influence in the development of academic activities and overall students&#39; performance. For example, stress levels are directly related to the onset of the burnout syndrome in students9,10,11, and high stress levels are especially relevant during the freshman year, where drop-out rates between 20% and 30%12,13 are common. Detecting and controlling stress indicators could dramatically improve academic performance.
The use of COTS wrist wearable devices is justified because they have sensors that provide information on physiological signals that have been widely used by the scientific community in stress assessment and detection. Some of the signals referred to in the literature used for this purpose include heart rate (HR)14, heart rate varaibility15, skin temperature (ST)16, respiration14, and galvanic skin response (GSR)17. These signals can be collected by COTS wrist wearables. However, they do not offer the same performance as clinical devices. There are differences related to the accuracy of sensors among devices18,19,20,21. Nevertheless, previous works18,19,20,21 have shown that, in a slow movement scenario, COTS wrist wearable sensors have error patterns similar to specialized devices.
The aim of this paper is to introduce a protocol to evaluate different solutions for stress estimation in students using COTS wrist wearables. There are many arrangements that can be proposed to estimate stress levels, involving the use of different wrist wearable devices and data analytics techniques, and more specifically machine learning algorithms. COTS wrist wearables are characterized by their high fragmentation, heterogeneity and interoperability problems22. Three companies have an aggregated market share of almost 50%23, but many other companies account for much smaller individual market shares, with an aggregated share above 50%. On the other hand, in terms of heterogeneity, not all wearables have the same number and type of sensors, with accelerometers and h sensors being the most common, and ST&#39;s and GSR&#39;s being only present in 5% of the devices studied. As for interoperability, there are different operating systems and data collection approaches that are not compatible with each other. As for the machine learning techniques that can be applied to estimate stress from the data collected by means of a wrist device, there are many options available24, including decision trees, neural networks, nearest neighbor approaches, Na&#239;ve Bayes classifiers, etc. To sum up, there is a great variety of solutions that may be developed for stress estimation, so it is instrumental to design an evaluation protocol to facilitate the comparison among different tentative options to eventually select the most suitable in a given context.
For the implementation of the protocol, several tools are needed (Figure 1). First, a COTS wrist wearable device is needed to fetch physiological data. This wearable device should have at least HR monitoring capabilities, but additional sensors are desirable (e.g., accelerometer, ST, GSR sensors). Second, a smartphone running the PhysiologicalSignal app is required to collect the data captured by the wearable device. Third, a tablet running the StressTest app is needed to run stress induction exercises (the smartphone could be used instead the tablet for this purpose). Fourth, some questionnaires to collect qualitative data on students&#39; perception on stress. Fifth, a server with a Web service25 to perform data collection and pre-processing, and a Web dashboard to show the evolution of the signals. And finally, a data analytics package26 to process the data collected about students using machine learning techniques.
The evaluation protocol is organized into two phases. The first one, the laboratory phase, is carried out in a comfortable room, where different stress levels (i.e., &#34;relax&#34;, &#34;concentrated stress&#34; and &#34;stress&#34;) are induced to a target subject (a student) through several common stress-inducing tasks. The second part takes place in the classroom, and it involves monitoring the student during the accomplishment of several academic activities: theoretical explanations, individual activities, short tests, exams, etc. During the implementation of this protocol, the subject&#39;s physiological signals are captured by means of a wrist device. Finally, these signals are processed by machine learning algorithms to provide estimations on the level of stress.
During the laboratory phase, the StressTest app is used to induce different stress levels. This app guides the subject to the completion of four different tasks. The first task is to create a baseline for stress analysis. In this task, the student visualizes a 4-minute relaxing video in which different shots of a sunset on a bridge are shown. The second task is an adaptation of the Stroop Color and Word Test27&#160;(SCWT). Every two seconds, the subject must choose the color in which the name of a color is painted (red, green, orange, blue and purple). Several buttons located at the bottom of the screen containing the initial letter of each color are available for the subject to choose the painted color at each time. For example, the button that refers to blue depicts the letter B. In our case, this test is divided into three different levels of difficulty. For the first level (SCWT1), the colored &#34;words of colors&#34; will appear in the same order as the buttons, so color and name match directly. This level is taken as baseline, as it does not involve any difficulty and the subject should only press the buttons properly, always in the same order. For the second level (SCWT2), the colored &#34;words of colors&#34; appear randomly, but the correspondence between name and color is maintained. Every time the subject fails a beep is emitted, and if two errors are made, the correct color score will be reset. For the last, most difficult level (SCWT3), name and color do not match. In this way this level is intended to be more complex and stressful for the subject. The third task consists on the Paced Auditory Serial Addition test (PASAT)28, which measures how the student experiences a concentration test. During this task, a sequence of consecutive numbers is played aloud, and the student must add the last two numbers and write the result in the provided on-screen box before listening to the next number. In this task, if the subject makes a mistake, a disturbing event occurs to generate stress (two numbers sound at the same time or a long period of silence in maintained). In this case, if three errors are committed, the sum account will be reset. The fourth task consists on a hyperventilation activity to induce the same variation in the physiological signals that would provoke a stressful situation17. At the end of each task and level, the subject has to indicate the level of perceived stress, using the application itself, according to a 5-value Likert scale.
During the classroom phase, students carry out their ordinary academic activities together with the rest of their classmates. The protocol focuses on the stress levels that occur during classroom-specific activities. At the end of the lecture, a brief questionnaire (Annex 1) is completed by the student to indicate the perceived level of stress in the several activities according to a 5-value scale.

<table><tbody><tr><td>Microsoft Band 2</td><td>Microsoft</td><td></td><td>Wearable</td></tr><tr><td>Nexus 5</td><td>LG/Google</td><td></td><td>Smartphone</td></tr><tr><td>PhysiologicalSignals</td><td>Developed by the paper authors</td><td></td><td>App to collect data from wearables</td></tr><tr><td>StressTest</td><td>Developed by the paper authors</td><td></td><td>App to develop laboraty experiment: Video visualization, Stroop color and word test, paced auditory serial addition test (PASAT) , hyperventilation activity&amp;nbsp;</td></tr><tr><td>Quizs</td><td>Developed by the paper authors</td><td></td><td>Questionnaires to collect qualitative data on students&amp;rsquo; perception on stress</td></tr><tr><td>Server</td><td>Developed by the paper authors</td><td></td><td>Server to store, analyze and display data</td></tr><tr><td>Weka application</td><td>University of Waikato</td><td></td><td>Application to process the data using machine learning techniques</td></tr></tbody></table>

evaluation of commercial-off-the-shelf wrist wearables to estimate stress on students

Wearable commercial-off-the-shelf (COTS) devices have become popular during the last years to monitor sports activities, primarily among young people. These devices include sensors to gather data on physiological signals such as heart rate, skin temperature or galvanic skin response. By applying data analytics techniques to these kinds of signals, it is possible to obtain estimations of higher-level aspects of human behavior. In the literature, there are several works describing the use of physiological data collected using clinical devices to obtain information on sleep patterns or stress. However, it is still an open question whether data captured using COTS wrist wearables is sufficient to characterize the learners' psychological state in educational settings. This paper discusses a protocol to evaluate stress estimation from data obtained using COTS wrist wearables. The protocol is carried out in two phases. The first stage consists of a controlled laboratory experiment, where a mobile app is used to induce different stress levels in a student by means of a relaxing video, a Stroop Color and Word test, a Paced Auditory Serial Addition test, and a hyperventilation test. The second phase is carried out in the classroom, where stress is analyzed while performing several academic activities, namely attending to theoretical lectures, doing exercises and other individual activities, and taking short tests and exams. In both cases, both quantitative data obtained from COTS wrist wearables and qualitative data gathered by means of questionnaires are considered. This protocol involves a simple and consistent method with a stress induction app and questionnaires, requiring a limited participation of support staff.

The protocol discussed was put into practice in a Computer Architectures course in the first year of the Telecommunication Engineering degree at the University of Vigo. This course has more than 200 students enrolled who are organized into 10 working groups. To carry out this experiment, students from four of the groups were invited to enroll at the beginning of the academic year. The project attracted considerable interest among the students, and around 30 students volunteered to participate in the study. From them, 12 students were randomly selected for participation.
The COTS wrist wearable device selected for our experiments has HR, ST, GSR and accelerometer sensors. The choice of this wearable was based on its variety of sensors and the provision of real-time data feeding. Technical conditions in which sensor data is collected were also taken into account. Data capture is performed at certain frequencies, generally imposed by the operation of the sensors, but also due to the device&#39;s energy-saving characteristics. In the case of the selected device, HR was sampled every second (1 Hz). The accelerometer offered 62 Hz, 31 Hz and 8 Hz as sampling frequencies, from which 8 Hz was selected because it offers enough granularity for movement capture with reasonable energy requirements when compared to the other frequencies. GSR may be sampled at 0.2 or 5 Hz. In this case, we opted to gather GSR data once every 5 seconds. As for the accelerometer, this frequency provided enough granularity while keeping energy requirements to a minimum. Finally, ST is sampled at the same frequency as HR (i.e., 1 Hz). Data collected by the device is transferred to the PhysiologicalSignals app in the smartphone every second, including the HR and ST sample, the maximum acceleration value, and the last value for GSR collected. To reduce HR noise, the server applies to the received data a FIR filter commonly used in real-time applications29 and in the filtering of ECG signals30, using a 15-sample window.
Information gathered during laboratory and classroom sessions is stored in the server&#180;s database. This information should be downloaded to be processed using a data analytics package. The set of generated data files contains raw signals&#39; data and variables derived from those signals. More specifically, for each raw physiological signal (HR, ST, GSR and accelerometer), its standard deviation (st), slope (sl), and the difference between the present value and the extreme value in the last 30 seconds are recorded.
The laboratory phase of the protocol was carried out in a comfortable room of the Telematics Engineering department that has the appropriate conditions for the experiment. Figure 2 depicts the evolution of HR, GSR and ST values collected during one of these sessions for an actual student. As can be seen in the figure, significant variations in the physiological signals occur as the student performs each of the tasks (video, STC1, STC2, STC3, PASAT and Hyperventilation) included in the experiment. A relatively high initial HR value can be observed, most probably due to the stress induced when facing this task for the first time while being monitored. The rapid growth of ST during the hyperventilation test is also noteworthy.
Also observed during the laboratory experiments were the remarkable variations in the physiological signals at specific experimental moments, no matter that these periods were not always perceived as stressful by the target student. This is due to the fact that perceived stress is a subjective variable, and participating students do not fully agree in a common concept of stress. During the laboratory phase, it was intended to generate brief periods of high stress. These brief periods of stress were sometimes defined as frustration, but not as stress, which leads participating students to respond differently to what their physiological signals expressed. This effect can be visualized in the graphs in Figure 3. For example, in the interval between 12:15 and 12:20 (completion of the last test of the Stroop Color and Word Test) the strong GSR variations are a clear symptom of potential stress. These strong variations are also present between 12:25 and the end of the test (Hyperventilation test), but on both occasions, the user claimed to feel a similarly low stress level.
The situation discussed above stresses the subjective character of stress evaluation in such a short period of time. As a consequence, from the candidates for dependent variables in data sets (i.e., activity type, activity-based stress, or subject-perceived stress) we opted for activity-based stress. This variable defines stress levels according to the level of difficulty of the task addressed and not on the answers provided by the students about their perceived stress levels at the end of each task. This way, video watching would be tagged as &#34;relax&#34; while SCWT3 and PASAT would be labelled as &#34;concentration&#34; and the Hyperventilation test as &#34;stress&#34;. Note that samples from SCWT1 and SCWT2 were discarded in our case because in a previous pilot research was observed that, on average, SCWT1 and SCWT2 are activities that show a transition between a relaxed feeling (reached during video visualization) and stressful one. For these reason, we discarded from our analysis the signals from these 2 activities, and we included only those from video visualization, SCWT3, PASAT and Hyperventilation activities. The HR, ST and GSR variations among these states (relax, concentration, stress) are summarized in Figure 4. This figure depicts the physiological signal quartiles for the three stress levels in the 12 students involved in the experiment. In general, HR and GSR signals gradually increase as the student faces tasks of increasing difficulty. Also, in all cases the temperature level is affected. However, in some cases it increases for relaxed events and decreases in stressful situations, while in other cases it occurs just the opposite depending of the person.
In order to analyze the correlation observed visually in the variation of the physiological signals, machine learning techniques were applied over processed CSV files. To avoid initial transitory variations for each task and level, only the last 3 minutes of each activity are considered in order to avoid non-representative samples. In particular, several classification algorithms, particularly SVM, C4.5, k-NN, Random Forest, NaiveBayes and ZeroR, were trained to detect stress situations from the collected physiological signals. The trained classifiers became high accuracy, low mean absolute error rates and high Cohen&#39;s Kappa index level stress detectors, as it is shown in Table 1. For all the 12 subjects and algorithms (except ZeroR), the accuracy of stress detection in over 90%, mean absolute error value is near 0 and Cohen&#39;s kappa index is close to 1.
The classroom phase defined in the protocol took place during actual course sessions in the lecture rooms of the School of Telecommunications Engineering. Several academic activities were considered for this study: theoretical lectures; questions arbitrarily asked by the teacher to the students about some aspect of the course; doubts or questions posed to the teacher by students; short tests; regular exams/finals consisting of collection of problems to be solved by the student in 50-70 minutes.
The visualization of the evolution of physiological signals in this case shows that variations are subtler, that is, the differences in signal values for different activities are smaller than during the laboratory phase. The most relevant variations were observed during classroom sessions in which a regular lecture occurs after a pop quiz is completed. In this case, one or several of the physiological signals suffer significant differences, as illustrated in Figure 5. This figure depicts the signals captured for a student facing a short test (first part of the graphs). During the test, the most relevant variable would be HR. It can be observed that the student has a higher heart rate when compared to theoretical lecture time. In the same way, skin temperature is kept relatively low when compared to theoretical lecture time, when it raises around 1 &#176;C.
To analyze this in a numerical way, the correlation between the variations in the signals and the activities addressed by the students, machine learning techniques were applied analogously to the laboratory phase. The results for the combined pop quiz and lecture sessions show an average classification accuracy of 97.62% (&#177; 3.82) using C4.5. Note that for the analysis of these sessions skin temperature was discarded due to possible biases in the final result. During the transition period between the pop quiz and the following lecture students leave the classroom for approximately 20 minutes, with dramatically affects temperature values.
A comprehensive formal analysis of the collected classroom sessions is still in progress. This is a complex process where several challenging situations are addressed. First, abrupt short-time variations in the physiological signals are frequently observed with no associated stress-generating event. In most cases, these periods last for less than one minute without anything significant being recorded by the researcher. Another incidence observed is the instability of the GSR values when the wearable is not well adjusted or if sudden movements occur. Both situations result in a very low GSR values, close to 0 &#181;S. In a similar way, although much less usual, there are incorrect ST values, close to the ambient temperature, when the wearable is too big for the wrist of the user and therefore is loosely worn. To eliminate the analysis errors derived from these situations, affected variables are discarded. Note that all the signals monitored can be candidates to detect stress situations and different classifiers may be trained using different combinations of signals, but anomalous values would compromise classification no matter the classifier selected.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/57590/57590fig1.jpg" /> 
Figure 1. Tools used in the proposed protocol. This figure represents all the elements involved in the protocol and their interactions. <a href="/files/ftp_upload/57590/57590fig1large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/57590/57590fig2.jpg" /> 
Figure 2. Stress variation in a laboratory session. This figure shows the different parts in which the laboratory protocol is divided. Each part presents a clear variation in the physiological signals. <a href="/files/ftp_upload/57590/57590fig2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/57590/57590fig3.jpg" /> 
Figure 3. Stress variation perceived for a student in a laboratory session. This figure shows the discrepancies between the strong variations of the physiological signals of a student during a laboratory session and their answer to the stress quiz. <a href="/files/ftp_upload/57590/57590fig3large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/57590/57590fig4.jpg" /> 
Figure 4. Physiological signal percentiles for 12 students participating in a laboratory session. This figure represents a percentile summary for each subject. The strong physiological signal variations between each stress situation can be visualized. <a href="/files/ftp_upload/57590/57590fig4large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 5" class="xfigimg" src="/files/ftp_upload/57590/57590fig5.jpg" /> 
Figure 5. HR, ST and GSR variations during classroom activities. Physiological signals variation during a short test (Left). Physiological signals variation during a theoretical lecture (Right). <a href="/files/ftp_upload/57590/57590fig5large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<table fo:font-size="6" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td></td>
			<td colspan="3">SVM</td>
			<td colspan="3">C45</td>
			<td colspan="3">Knn</td>
			<td colspan="3">RandomForest</td>
			<td colspan="3">NaiveBayes</td>
			<td colspan="3">ZeroR</td>
		</tr>
		<tr>
			<td>StudentID</td>
			<td>Accuracy</td>
			<td>Error</td>
			<td>Kappa</td>
			<td>Accuracy</td>
			<td>Error</td>
			<td>Kappa</td>
			<td>Accuracy</td>
			<td>Error</td>
			<td>Kappa</td>
			<td>Accuracy</td>
			<td>Error</td>
			<td>Kappa</td>
			<td>Accuracy</td>
			<td>Error</td>
			<td>Kappa</td>
			<td>Accuracy</td>
			<td>Error</td>
			<td>Kappa</td>
		</tr>
		<tr>
			<td>100</td>
			<td>99.83</td>
			<td>0.22</td>
			<td>1</td>
			<td>99.75</td>
			<td>0</td>
			<td>1</td>
			<td>100</td>
			<td>0</td>
			<td>1</td>
			<td>100</td>
			<td>0</td>
			<td>1</td>
			<td>98.94</td>
			<td>0.01</td>
			<td>0.98</td>
			<td>43.12</td>
			<td>0.43</td>
			<td>0</td>
		</tr>
		<tr>
			<td>101</td>
			<td>98.77</td>
			<td>0.22</td>
			<td>0.98</td>
			<td>99.76</td>
			<td>0</td>
			<td>1</td>
			<td>99.81</td>
			<td>0</td>
			<td>1</td>
			<td>99.93</td>
			<td>0.01</td>
			<td>1</td>
			<td>98.16</td>
			<td>0.01</td>
			<td>0.97</td>
			<td>48.7</td>
			<td>0.42</td>
			<td>0</td>
		</tr>
		<tr>
			<td>102</td>
			<td>99.56</td>
			<td>0.22</td>
			<td>0.99</td>
			<td>99.83</td>
			<td>0</td>
			<td>1</td>
			<td>99.77</td>
			<td>0</td>
			<td>1</td>
			<td>99.91</td>
			<td>0.01</td>
			<td>1</td>
			<td>93.5</td>
			<td>0.05</td>
			<td>0.89</td>
			<td>52.41</td>
			<td>0.41</td>
			<td>0</td>
		</tr>
		<tr>
			<td>103</td>
			<td>99.71</td>
			<td>0.22</td>
			<td>1</td>
			<td>99.94</td>
			<td>0</td>
			<td>1</td>
			<td>99.97</td>
			<td>0</td>
			<td>1</td>
			<td>99.97</td>
			<td>0.01</td>
			<td>1</td>
			<td>97.24</td>
			<td>0.02</td>
			<td>0.96</td>
			<td>49.64</td>
			<td>0.42</td>
			<td>0</td>
		</tr>
		<tr>
			<td>104</td>
			<td>99.82</td>
			<td>0.22</td>
			<td>1</td>
			<td>99.33</td>
			<td>0.01</td>
			<td>0.99</td>
			<td>100</td>
			<td>0</td>
			<td>1</td>
			<td>99.85</td>
			<td>0.01</td>
			<td>1</td>
			<td>97.09</td>
			<td>0.02</td>
			<td>0.96</td>
			<td>42.05</td>
			<td>0.44</td>
			<td>0</td>
		</tr>
		<tr>
			<td>105</td>
			<td>100</td>
			<td>0.22</td>
			<td>1</td>
			<td>100</td>
			<td>0</td>
			<td>1</td>
			<td>99.84</td>
			<td>0</td>
			<td>1</td>
			<td>100</td>
			<td>0</td>
			<td>1</td>
			<td>99.83</td>
			<td>0</td>
			<td>1</td>
			<td>43.8</td>
			<td>0.43</td>
			<td>0</td>
		</tr>
		<tr>
			<td>106</td>
			<td>98.09</td>
			<td>0.23</td>
			<td>0.97</td>
			<td>99.37</td>
			<td>0.01</td>
			<td>0.99</td>
			<td>99.69</td>
			<td>0</td>
			<td>1</td>
			<td>99.85</td>
			<td>0.01</td>
			<td>1</td>
			<td>96.52</td>
			<td>0.02</td>
			<td>0.95</td>
			<td>47.51</td>
			<td>0.42</td>
			<td>0</td>
		</tr>
		<tr>
			<td>107</td>
			<td>100</td>
			<td>0.22</td>
			<td>1</td>
			<td>100</td>
			<td>0</td>
			<td>1</td>
			<td>99.85</td>
			<td>0</td>
			<td>1</td>
			<td>100</td>
			<td>0</td>
			<td>1</td>
			<td>99.96</td>
			<td>0</td>
			<td>1</td>
			<td>50.44</td>
			<td>0.42</td>
			<td>0</td>
		</tr>
		<tr>
			<td>108</td>
			<td>99.46</td>
			<td>0.22</td>
			<td>0.99</td>
			<td>99.76</td>
			<td>0</td>
			<td>1</td>
			<td>99.76</td>
			<td>0</td>
			<td>1</td>
			<td>100</td>
			<td>0</td>
			<td>1</td>
			<td>98.55</td>
			<td>0.01</td>
			<td>0.97</td>
			<td>59.76</td>
			<td>0.37</td>
			<td>0</td>
		</tr>
		<tr>
			<td>109</td>
			<td>99.54</td>
			<td>0.22</td>
			<td>0.99</td>
			<td>100</td>
			<td>0</td>
			<td>1</td>
			<td>99.78</td>
			<td>0</td>
			<td>1</td>
			<td>99.96</td>
			<td>0</td>
			<td>1</td>
			<td>99.78</td>
			<td>0</td>
			<td>1</td>
			<td>47.34</td>
			<td>0.42</td>
			<td>0</td>
		</tr>
		<tr>
			<td>110</td>
			<td>99.86</td>
			<td>0.22</td>
			<td>1</td>
			<td>99.94</td>
			<td>0</td>
			<td>1</td>
			<td>99.72</td>
			<td>0</td>
			<td>1</td>
			<td>99.9</td>
			<td>0.01</td>
			<td>1</td>
			<td>96.4</td>
			<td>0.02</td>
			<td>0.94</td>
			<td>50.35</td>
			<td>0.42</td>
			<td>0</td>
		</tr>
		<tr>
			<td>111</td>
			<td>99.97</td>
			<td>0.22</td>
			<td>1</td>
			<td>99.84</td>
			<td>0</td>
			<td>1</td>
			<td>100</td>
			<td>0</td>
			<td>1</td>
			<td>100</td>
			<td>0</td>
			<td>1</td>
			<td>99.35</td>
			<td>0</td>
			<td>0.99</td>
			<td>43.7</td>
			<td>0.43</td>
			<td>0</td>
		</tr>
		<tr>
			<td>Average</td>
			<td>99.55</td>
			<td>0.22</td>
			<td>0.99</td>
			<td>99.79</td>
			<td>0.00</td>
			<td>1.00</td>
			<td>99.85</td>
			<td>0.00</td>
			<td>1.00</td>
			<td>99.95</td>
			<td>0.01</td>
			<td>1.00</td>
			<td>97.94</td>
			<td>0.01</td>
			<td>0.97</td>
			<td>48.24</td>
			<td>0.42</td>
			<td>0.00</td>
		</tr>
		<tr>
			<td>Standard Deviation</td>
			<td>0.55</td>
			<td>0.00</td>
			<td>0.01</td>
			<td>0.22</td>
			<td>0.00</td>
			<td>0.00</td>
			<td>0.11</td>
			<td>0.00</td>
			<td>0.00</td>
			<td>0.06</td>
			<td>0.01</td>
			<td>0.00</td>
			<td>1.82</td>
			<td>0.01</td>
			<td>0.03</td>
			<td>4.72</td>
			<td>0.02</td>
			<td>0.00</td>
		</tr>
	</tbody>
</table>
Table 1. Accuracy, mean absolute error, and Cohen&#39;s Kappa index values obtained for SVM, C4.5, k-NN, Random Forest, NaiveBayes and ZeroR machine learning classifiers using data from the 12 students participating in the laboratory experiment.

Watch this Scientific Journal Video about Evaluation of Commercial-Off-The-Shelf Wrist Wearables to Estimate Stress on Students at JoVE.com

Evaluation of Commercial-Off-The-Shelf Wrist Wearables to Estimate Stress on Students

A protocol to evaluate solutions based on commercial-off-the-shelf (COTS) wrist wearables to estimate stress in students is proposed. The protocol is carried out in two phases, an initial laboratory-based stress induction test, and a monitoring stage taking place in the classroom while the student is performing academic activities.

Wearable commercial-off-the-shelf (COTS) devices have become popular during the last years to monitor sports activities, primarily among young people. ...

evaluation-commercial-off-shelf-wrist-wearables-to-estimate-stress-on

Nanyang Technological University

Research

JoVE Journal

Engineering

7.4K Views.  University of Vigo, Campus Lagoas-Marcosende. A protocol to evaluate solutions based on commercial-off-the-shelf (COTS) wrist wearables to estimate stress in students is proposed. The protocol is carried out in two phases, an initial laboratory-based stress induction test, and a monitoring stage taking place in the classroom while the student is performing academic activities.

Video: Evaluation of Commercial-Off-The-Shelf Wrist Wearables to Estimate Stress on Students

An Alternative to the Traditional Cold Pressor Test: The Cold Pressor Arm Wrap

Recently research on the relationship between stress and cognition, emotion, and behavior has greatly increased. These advances have yielded insights into important questions ranging from the nature of stress&#39; influence on addiction1 to the role of stress in neural changes associated with alterations in decision-making2,3. As topics being examined by the field evolve, however, so too must the methodologies involved. In this article a practical and effective alternative to a classic stress induction technique, the cold pressor test (CPT), is presented: the cold pressor arm wrap (CPAW). CPT typically involves immersion of a participant&#39;s dominant hand in ice-cold water for a period of time4. The technique is associated with robust activation of the sympatho-adrenomedullary (SAM) axis (and release of catecholamines; e.g.&#160;adrenaline and noradrenaline) and mild-to-moderate activation of the hypothalamic-pituitary-adrenal (HPA) axis with associated glucocorticoid (e.g.&#160;cortisol) release. While CPT has been used in a wide range of studies, it can be impractical to apply in some research environments. For example use of water during, rather than prior to, magnetic resonance imaging (MRI) has the potential to damage sensitive and expensive equipment or interfere with acquisition of MRI signal. The CPAW is a practical and effective alternative to the traditional CPT. Composed of a versatile list of inexpensive and easily acquired components, CPAW makes use of MRI-safe gelpacs cooled to a temperature similar to CPT rather than actual water. Importantly CPAW is associated with levels of SAM and HPA activation comparable to CPT, and can easily be applied in a variety of research contexts. While it is important to maintain specific safety protocols when using the technique, these are easy to implement if planned for. Creation and use of the CPAW will be discussed.

The cold pressor test is a classic stress induction task used in a wide range of research examining the relationship between stress and cognition, emotion, and behavior. In this article, a practical and versatile alternative to the traditional cold pressor test is presented: the cold pressor arm wrap.

Recently research on the relationship between stress and cognition, emotion, and behavior has greatly increased. These advances have yielded insights into ...

Behavior

Use of Galvanic Skin Responses, Salivary Biomarkers, and Self-reports to Assess Undergraduate Student Performance During a Laboratory Exam Activity

Typically, self-reports are used in educational research to assess student response and performance to a classroom activity. Yet, addition of biological and physiological measures such as salivary biomarkers and galvanic skin responses are rarely included, limiting the wealth of information that can be obtained to better understand student performance. A laboratory protocol to study undergraduate students&#39; responses to classroom events (e.g., exams) is presented. Participants were asked to complete a representative exam for their degree. Before and after the laboratory exam session, students completed an academic achievement emotions self-report and an interview that paralleled these questions when participants wore a galvanic skin sensor and salivary biomarkers were collected. Data collected from the three methods resulted in greater depth of information about students&#39; performance when compared to the self-report. The work can expand educational research capabilities through more comprehensive methods for obtaining nearer to real-time student responses to an examination activity.

Research on undergraduate students' academic achievement emotions have primarily relied on self-reports in laboratory settings. Studies rarely include bio-physiological measures to these self-reports. This protocol will describe a methodology that integrates self-reports with and bio-physiological measures to assess student response and performance during a laboratory examination activity.

Typically, self-reports are used in educational research to assess student response and performance to a classroom activity. Yet, addition of biological ...

Reducing State Anxiety Using Working Memory Maintenance

The purpose of this protocol is to explain how to examine the relationship between working memory processes and anxiety by combining the Sternberg Working Memory (WM) and the threat of shock paradigms. In the Sternberg WM paradigm, subjects are required to maintain a series of letters in the WM for a brief interval and respond by identifying whether the position of a given letter in the series matches a numerical prompt. In the threat of shock paradigm, subjects are exposed to alternating blocks where they are either at risk of receiving unpredictable presentations of a mild electric shock or are safe from the shock. Anxiety is probed throughout the safe and threat blocks using the acoustic startle reflex, which is potentiated under threat (Anxiety-Potentiated Startle (APS)). By conducting the Sternberg WM paradigm during the threat of shock and probing the startle response during either the WM maintenance interval or the intertrial interval, it is possible to determine the effect of WM maintenance on APS.

This protocol demonstrates how to measure anxiety-potentiated startle during the Sternberg Working Memory paradigm.

The purpose of this protocol is to explain how to examine the relationship between working memory processes and anxiety by combining the Sternberg Working ...

A Community-based Stress Management Program: Using Wearable Devices to Assess Whole Body Physiological Responses in Non-laboratory Settings

A pragmatic breath-based intervention to benefit human performance and stress management is timely and valuable to individuals seeking holistic approaches for emotional regulation and optimizing compensatory reserve mechanisms. This protocol is designed to not only teach mind-body awareness but also to provide feedback utilizing physiological data and survey results. The primary findings of this study showed that heart coherence and alpha variables were significantly correlated after four weeks of the breath-based meditation stress protocol. Meditation and rhythmic breathing produced significant increases in alpha brain activity. These brain physiological responses conformed to the Pleth Variability Index (PVI) changes, suggesting the ability of the human body to enter into a meditative state and effectively manage stress. When assessed after four weeks of daily practicing the techniques employed in the stress management protocol, based on the Five Facet Mindfulness Questionnaire, subjects improved in applying mindfulness skills. The overall mindfulness score, Pleth variability index (PVI), and perfusion index (PI) increased after the 4-week intervention period. Results from electroencephalography (brain waves) were consistent with a meditative state during the post-study follow-up session. This provides evidence that wearable devices are feasible for data collection during a breath-based stress management intervention. This protocol can be easily and efficiently implemented into any study design in which physiological data are desired in a non-laboratory-based setting.

Stress is an unavoidable and persistent component of life and holistic approaches for its management are being considered. A standardized methodology was created to demonstrate the feasibility of a breath-based stress management protocol that can be used with commercially available portable technology.

A pragmatic breath-based intervention to benefit human performance and stress management is timely and valuable to individuals seeking holistic approaches ...

An Application for Pairing with Wearable Devices to Monitor Personal Health Status

The current protocol aims to showcase the technology integration, providing a detailed description of adopting the HealthCloud app, developed by the Healthy Landscape and Healthy People Lab, National Taiwan University (HLHP-NTU), on smartphones and smartwatches to collect data on users' real-time psychological and physiological responses and environmental information. A flexible and integrated research method was proposed because it can be difficult to measure multi-dimensional aspects of personal data in on-site studies in landscape and outdoor recreation research. An on-site study conducted in 2020 at the National Taiwan University campus was used as an application example. A dataset of 385 participants was used after excluding invalid samples. During the experiment, participants were asked to walk around campus for 30 min when their heart rate and psychological-scale items were measured, together with several environmental metrics. This work aimed to provide a possible solution to help on-site studies track real-time human responses that match ambient factors. Due to the app's flexibility, its use on wearable devices shows excellent potential for multidisciplinary research studies.

The present protocol introduces a non-commercial self-developed application for collecting real-time on-site data, including psychological scales, GPS location, heart rate, and blood-oxygen saturation level, as well as the application's operating procedures. An empirical study conducted in Taiwan in 2020 was used as an application example.

The current protocol aims to showcase the technology integration, providing a detailed description of adopting the HealthCloud app, developed by the ...

Environment

Psychophysiological Stress Assessment Using Biofeedback 

In the last half century, research in biofeedback has shown the extent to which the human mind can influence the functioning of the autonomic nervous system, previously thought to be outside of conscious control. By letting people observe signals from their own bodies, biofeedback enables them to develop greater awareness of their physiological and psychological reactions, such as stress, and to learn to modify these reactions. Biofeedback practitioners can facilitate this process by assessing people s reactions to mildly stressful events and formulating a biofeedback-based treatment plan. During stress assessment the practitioner first records a baseline for physiological readings, and then presents the client with several mild stressors, such as a cognitive, physical and emotional stressor. Variety of stressors is presented in order to determine a person's stimulus-response specificity, or differences in each person's reaction to qualitatively different stimuli. This video will demonstrate the process of psychophysiological stress assessment using biofeedback and present general guidelines for treatment planning.

Psychophysiological Stress Assessment Using Biofeedback

In this video we will describe one method of assessing person's psychophysiological reaction to stress using biofeedback. We will also present general guidelines for treatment planning.

In the last half century, research in biofeedback has shown the extent to which the human mind can influence the functioning of the autonomic nervous ...

Biology

A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences

Over the past ten years, research into students&#39; emotions in educational environments has increased. Although researchers have called for more studies that rely on objective measures of emotional experience, limitations on utilizing multi-modal data sources exist. Studies of emotion and emotional regulation in classrooms traditionally rely on survey instruments, experience-sampling, artifacts, interviews, or observational procedures. These methods, while valuable, are mainly&#160;dependent on participant or observer subjectivity and is limited in its authentic measurement of students&#39; real-time performance to a classroom activity or task. The latter, in particular, poses a stumbling block to many scholars seeking to objectively measure emotions and other related measures in the classroom, in real-time.
The purpose of this work is to present a protocol to experimentally study students&#39; real-time responses to exam experiences during an authentic assessment situation. For this, a team of educational psychologists, engineers, and engineering education researchers designed an experimental protocol that retained the limits required for accurate physiological sensor measurement, best-practices of salivary collection, and an authentic testing environment. In particular, existing studies that rely on physiological sensors are conducted in experimental environments that are disconnected from educational settings (e.g., Trier Stress Test), detached in time (e.g., before or after a task), or introduce analysis error (e.g., use of sensors in environments where students are likely to move). This limits our understanding of students&#39; real-time responses to classroom activities and tasks. Furthermore, recent research has called for more considerations to be covered around issues of recruitment, replicability, validity, setups, data cleaning, preliminary analysis, and particular&#160;circumstances (e.g., adding a variable in the experimental design) in academic emotions research that relies on multi-modal approaches.

An experimental design was developed to investigate the real-time influences of an examination experience to assess the emotional realities students experience in higher education settings and tasks. This design is the result of a cross-disciplinary (e.g., educational psychology, biology, physiology, engineering) and multi-modal (e.g., salivary markers, surveys, electrodermal sensor) approach.

Over the past ten years, research into students&#39; emotions in educational environments has increased. Although researchers have called for more studies ...

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios

In medicine or industry, the analysis of high-dimensional data sets is increasingly required. However, available technical solutions are often complex to use. Therefore, new approaches like immersive analytics are welcome. Immersive analytics promise to experience high-dimensional data sets in a convenient manner for various user groups and data sets. Technically, virtual-reality devices are used to enable immersive analytics. In Industry 4.0, for example, scenarios like the identification of outliers or anomalies in high-dimensional data sets are pursued goals of immersive analytics. In this context, two important questions should be addressed for any developed technical solution on immersive analytics: First, is the technical solutions being helpful or not? Second, is the bodily experience of the technical solution positive or negative? The first question aims at the general feasibility of a technical solution, while the second one aims at the wearing comfort. Extant studies and protocols, which systematically address these questions are still rare. In this work, a study protocol is presented, which mainly investigates the usability for immersive analytics in Industry 4.0 scenarios. Specifically, the protocol is based on four pillars. First, it categorizes users based on previous experiences. Second, tasks are presented, which can be used to evaluate the feasibility of the technical solution. Third, measures are presented, which quantify the learning effect of a user. Fourth, a questionnaire&#160;evaluates the stress level when performing tasks. Based on these pillars, a technical setting was implemented that uses mixed reality smartglasses to apply the study protocol. The results of the conducted study show the applicability of the protocol on the one hand and the feasibility of immersive analytics in Industry 4.0 scenarios on the other. The presented protocol includes a discussion of discovered limitations.

This protocol delineates the technical setting of a developed mixed reality application that is used for immersive analytics. Based on this, measures are presented, which were used in a study to gain insights into usability aspects of the developed technical solution.

In medicine or industry, the analysis of high-dimensional data sets is increasingly required. However, available technical solutions are often complex to ...

Setup of Consumer Wearable Devices for Exposure and Health Monitoring in Population Studies

Wearable sensors, which are often embedded in commercial smartwatches, allow for continuous and non-invasive health measurements and exposure assessments in clinical studies. Nevertheless, the real-life application of these technologies in studies involving a large number of participants for a significant observation period may be hindered by several practical challenges. 
In this study, we present a modified protocol from a previous intervention study for the mitigation of health effects from desert dust storms. The study involved two distinct population groups: asthmatic children aged 6-11 years and elderly patients with atrial fibrillation (AF). Both groups were equipped with a smartwatch for the assessment of physical activity (using a heart rate monitor, pedometer, and accelerometer) and location (using GPS signals to locate individuals in indoor "at home" or outdoor microenvironments). The participants were required to wear the smartwatch equipped with a data collection application on a daily basis, and data were transmitted via a wireless network to a centrally administered data collection platform for the near real-time assessment of compliance. 
Over a period of 26 months, more than 250 children and 50 patients with AF participated in the aforementioned study. The main technical challenges identified included restricting access to standard smartwatch features, such as gaming, internet browser, camera, and audio recording applications, technical issues, such as loss of GPS signal, especially in indoor environments, and the internal smartwatch settings interfering with the data collection application.
The aim of this protocol is to demonstrate how the use of publicly available application lockers and device automation applications made it possible to address most of these challenges in a simple and cost-effective way. In addition, the inclusion of a Wi-Fi received signal strength indicator significantly improved indoor localization and largely minimized GPS signal misclassification. The implementation of these protocols during the roll-out of this intervention study in the spring of 2020 led to significantly improved results in terms of data completeness and data quality.

Commercial smartwatches equipped with wearable sensors are increasingly being used in population studies. However, their utility is often constrained by their limited battery duration, memory capacity, and data quality. This report provides examples of cost-effective solutions to real-life technical challenges encountered during studies involving asthmatic children and elderly cardiac patients.

Wearable sensors, which are often embedded in commercial smartwatches, allow for continuous and non-invasive health measurements and exposure assessments ...

Bioengineering

Evaluation of Hydration Status by Bioelectrical Impedance Vector Analysis in Patients with Ischemic Heart Disease Undergoing Exercise Stress Test

Ischemic heart disease (IHD) represents a group of clinical syndromes characterized by myocardial ischemia, leading to an impairment in the myocardial blood supply and compromised perfusion. Several clinical variables assessed through a stress test, such as oxygen uptake (VO2) and heart rate oxygen pulse (HR/O2), have been attributed as cardiopulmonary prognostic factors in patients with IHD. However, other factors like hydration status (HS), potentially affecting the cardiopulmonary response, have been barely addressed. Unbalanced HS has a short-term effect on plasma volume and the sympathetic nervous system, which impacts blood volume, and lowers VO2 and HR/O2. Recently, bioelectrical impedance analysis (BIA), a method based on the opposition of body tissues (including fluid volume) to a low electrical current, has been widely used to assess HS by obtaining two components: resistance (R) and reactance (Xc) and using prediction formulas. However, several limitations as chronic illness or abnormal fluid status, may affect the results. In this sense, alternative BIA methods, such as bioelectrical impedance vector analysis (BIVA), have become relevant. R and Xc (adjusted by height) result in a vector plotted on the R/Xc graph, which allows interpreting the HS as normal or abnormal according to the distance of the mean vector. This study aims to describe how to determine HS by BIVA using a single-frequency device and compare the results with the cardiopulmonary response in patients with IHD.

Hydration status imbalances can have a short-term effect on direct and indirect determinants of oxygen uptake and pulse, and morbidity and mortality prognostic factors in ischemic heart disease. This protocol describes the technique for assessment of hydration status through bioelectrical impedance vector analysis and cardiopulmonary response during exercise stress test.

Ischemic heart disease (IHD) represents a group of clinical syndromes characterized by myocardial ischemia, leading to an impairment in the myocardial ...

Evaluation of Commercial-Off-The-Shelf Wrist Wearables to Estimate Stress on Students

Summary

Explore More Videos

Chapters in this video

An Alternative to the Traditional Cold Pressor Test: The Cold Pressor Arm Wrap

Use of Galvanic Skin Responses, Salivary Biomarkers, and Self-reports to Assess Undergraduate Student Performance During a Laboratory Exam Activity

Reducing State Anxiety Using Working Memory Maintenance

A Community-based Stress Management Program: Using Wearable Devices to Assess Whole Body Physiological Responses in Non-laboratory Settings

An Application for Pairing with Wearable Devices to Monitor Personal Health Status

Psychophysiological Stress Assessment Using Biofeedback

A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios

Setup of Consumer Wearable Devices for Exposure and Health Monitoring in Population Studies

Evaluation of Hydration Status by Bioelectrical Impedance Vector Analysis in Patients with Ischemic Heart Disease Undergoing Exercise Stress Test