This project was funded, in part, by ConAgra Foods (Omaha, NE, USA), the Virginia Agricultural Experiment Station, the Hatch Program of the National Institute of Food and Agriculture, U.S. Department of Agriculture, and the Virginia Tech Water INTERface Interdisciplinary Graduate Education Program.

Ethics Statement: This study was pre-approved by Virginia Tech Institutional Review Board (IRB) (IRB 14-229) prior to starting the project.
Caution: Human subject research requires informed consent prior to participation. In addition to IRB approval, consent for use of still or video images is also required prior to releasing any images for print, video, or graphic imaging. Additionally, food allergens are disclosed prior to testing. Participants are asked prior to panel start if they have any intolerance, allergies or other concerns.
Note: Exclusion Criteria: Automated facial expression analysis is sensitive to thick framed glasses, heavily bearded faces and skin tone. Participants who have these criteria are incompatible with software analysis due to an increased risk of failed videos. This is attributed to the software&#39;s inability to find the face.
1. Sample Preparation and Participant Recruitment
<ol>
	<li>Prepare beverage or soft food samples.
	<ol>
		<li>Prepare intensified dairy solutions using 2% milk and suggested flavors from Costello and Clark (2009)22 as well as other flavors. Prepare the following solutions: (1) unflavored milk (2% reduced fat milk); (2) unflavored water (drinking water); (3) vanilla extract flavor in milk (0.02 g/ml) (imitation clear vanilla flavor); and (4) salty flavor in milk (0.004 g/ml iodized salt). 
		Note: These solutions are used for demonstration purposes only.</li>
		<li>Pour half ounce aliquots (~15 g) of each solution into 2 oz. transparent plastic sample cups and cap with color coded lids. 
		Note: It is recommended to use transparent cups; however, it is up to the researcher&#39;s discretion.</li>
	</ol>
	</li>
	<li>Recruit participants from the campus or the local community to participate in the study. 
	Note: Participant sample size needed for a study is up to the discretion of the researcher. We recommend a range of 10 to 50 participants.</li>
	<li>Obtain human subject consent prior to participation in the study.</li>
</ol>
2. Preparation of Panel Room for Video Capture
Note: This protocol is for data capture in a sensory evaluation laboratory. This protocol is to make AFEA data capture useful for a sensory booth setting.
<ol>
	<li>Use individual booths with a touchscreen monitor in front of them (face level) to keep their focus forward and to prevent looking down.</li>
	<li>Use adjustable height chairs with back support. 
	Note: These are essential for allowing participants to be vertically adjusted and placed in a suitable range for video capture. Use stationary chairs (no rolling feature) with adjustable back height support so the participant&#39;s movements are reduced.</li>
	<li>Set overhead lighting at &#34;100% daylight&#34; for optimal facial emotional video capture (Illuminant 6504K; R=206; G=242; B=255). 
	Note: To avoid intense shadowing, diffuse frontal lighting is ideal while the light intensity or color is not as relevant20. Ultimately, it is up to the discretion of the researcher, individual protocol/methodology, and environment to control lighting for capture.</li>
	<li>Affix an adjustable camera above the touchscreen monitor for recording.
	<ol>
		<li>Use a camera with a resolution of at least 640 x 480 pixels (or higher)20. Discuss the required camera capabilities with the software provider before purchase and installation20. Note: The aspect ratio is not important20.</li>
		<li>Set camera capture speed to 30 frames per second (or other standard speed) for consistency.</li>
		<li>Connect and ensure media recording software is set up to the camera to record and save participant videos.</li>
	</ol>
	</li>
</ol>
3. Participant Adjustment and Verbal Directions
<ol>
	<li>Have only one participant at a time evaluate the samples in the sensory booth. 
	Note: Testing more than one participant at the same time may interfere with the testing environment and disrupt the concentration of the participant or create bias.</li>
	<li>Upon arrival, give participants verbal instructions about the process and standard operating procedures.
	<ol>
		<li>Have the participants sit straight up and against the back of the chair.</li>
		<li>Adjust chair height, position of the chair (distance from the camera), and camera angle so that the participant&#39;s face is captured in the center of the video recording, with no shadows on chin or around eyes. 
		Note: In the sensory booth, the participant&#39;s head is roughly 20 - 24 inches away from the camera and the monitor with the face centered in the camera video feed.</li>
		<li>Instruct participants to remain seated as positioned and focused facing towards the monitor display. Additionally, instruct participants to refrain from any sudden movements post-sample consumption during the 30 sec evaluation period per sample.</li>
		<li>Instruct the participant to consume the entire beverage or liquefied food sample and swallow.</li>
		<li>Instruct the participant to quickly move the sample cup below the chin and down to the table immediately after the sample is in the mouth. This is to eliminate facial occlusion. Remind them to keep looking toward the monitor. 
		Note: The sample carrier to deliver the sample is up to the discretion of the researcher. A straw or cup may be used. Regardless, initial facial occlusion is unavoidable because the face will be occluded or distorted due to consumption.</li>
	</ol>
	</li>
	<li>Instruct the participant to follow the instructions as they appear on the touchscreen monitor. Note: Instructions are automatically sequenced as programmed into the automated sensory software.</li>
</ol>
4. Individual Participant Process for Video Capture
<ol>
	<li>Confirm video camera is optimally capturing participant&#8217;s face while the participant is seated comfortably in the booth (before sample presentation) by viewing the computer monitor on which the video capture is displayed. Begin recording by clicking the record button on the computer monitor.</li>
	<li>Instruct participants to sip water to cleanse their palate.</li>
	<li>Provide treatments one at a time, starting with a baseline or control treatment (unflavored water). Identify each sample by a unique colored index card placed on top of each sample relating to the sample color code for sample treatment identification within the video. 
	Note: Programmed guidance on the touchscreen monitor instructs participants. The instructions direct the participant through a series of standardized steps for each treatment sample.</li>
	<li>Via the touchscreen monitor, direct the participant to:
	<ol>
		<li>Hold up the associated color index card pre-consumption for sample identification in the video. 
		Note: The color card is a way researchers can identify treatments in the video and mark the appropriate time frame (time zero) for sample evaluation.</li>
		<li>After holding the card briefly, place the card back on the tray.</li>
		<li>Fully consume the sample and wait approximately 30 seconds, enforced through the programmed guidance on the monitor, while facing towards the camera. 
		Note: The 30 sec controlled sampling period encompasses a time span adequate for the entire sampling evaluation period (i.e., showing the index card, opening a sample (removing the lid), consumption, and emotional capture).</li>
		<li>Enter their hedonic acceptability score on the touchscreen monitor (1=dislike extremely, 2=dislike very much, 3=dislike moderately, 4=dislike slightly, 5=neither like nor dislike, 6=like slightly, 7=like moderately, 8=like very much, 9=like extremely).</li>
		<li>Rinse mouth with drinking water before the next sample process.</li>
	</ol>
	</li>
</ol>
5. Evaluating Automated Facial Expression Analysis Options
Note: Many facial expression analysis software programs exist. Software commands and functions may vary. It is important to follow the manufacturer&#39;s user guidelines and reference manual20.
<ol>
	<li>Save recordings in a media format and transfer to the automated facial expression analysis software.</li>
	<li>Analyze participant videos using automated facial analysis software.
	<ol>
		<li>Double click on the software icon on the computer desktop.</li>
		<li>Once the program is open, click &#34;File&#34;, select &#34;New&#8230;&#34;, and select &#34;Project&#8230;&#34;</li>
		<li>In the pop up window, name the project and save the project.</li>
		<li>Add participants to the project by clicking the &#34;Add participants&#34; icon (Person with a (+) sign). More participants can be added by repeating this step.</li>
		<li>Add participant&#39;s video to the respective participant for analysis.
		<ol>
			<li>On the left side of the screen click the icon of the film reel with a plus (+) sign to add a video to analyze.</li>
			<li>Click the &#34;magnifying glass&#34; under the participant of interest to browse the video to add.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Analyze videos frame-by-frame under continuous calibration analysis settings in the software.
	<ol>
		<li>Click the pencil icon to adjust settings at the bottom of the window, under the &#34;settings&#34; tab for each participant video.
		<ol>
			<li>Set &#34;Face Model&#34; to General. Set &#34;Smoothen classifications&#34; to Yes. Set &#34;Sample Rate&#34; to Every frame.</li>
			<li>Set &#34;Image rotation&#34; to No. Set &#34;Continuous calibration&#34; to Yes. Set &#34;Selected calibration&#34; to None.</li>
		</ol>
		</li>
		<li>Save project settings.</li>
		<li>Press the batch analysis icon (the same red and black target-like symbol) to analyze the project videos.</li>
		<li>Save the results once analysis is completed. 
		Note: Other video settings exist in the software if researcher preference warrants another analysis method.</li>
		<li>Consider videos failures if serious facial occlusions or the inability to map the face persists during the specified post-consumption window (Figure 1). Additionally, if the model fails data will say &#34;FIT_FAILED&#34; or &#34;FIND_FAILED&#34; in the exported output files (Figure 2). This represents lost data since the software cannot classify or analyze the participant&#39;s emotions. 
		Note: AFEA translates facial muscle motion to neutral, happy, disgusted, sad, angry, surprised and scared on a scale from 0 (not expressed) to 1 (fully expressed) for each emotion.</li>
	</ol>
	</li>
	<li>Export the AFEA data output as log files (.txt) for further analysis.
	<ol>
		<li>Once analyses are complete, export the whole project.
		<ol>
			<li>Click &#34;File&#34;, &#34;Export&#34;, &#34;Export Project Results&#34;.</li>
			<li>When a window opens, choose the location of where the exports should be saved and save the log files (.txt) to a folder.</li>
			<li>Convert each participant log life to a data spreadsheet (.csv or .xlsx) to extract relevant data.
			<ol>
				<li>Open data spreadsheet software and select the &#34;Data&#34; tab.</li>
				<li>On the &#34;Data&#34; tab, in the &#34;Get External Data&#34; group, click &#34;From Text&#34;.</li>
				<li>In the &#34;Address bar&#34;, locate, double-click the participant text file to import, and follow the on screen wizard instructions.</li>
				<li>Continue the export process for all relevant participant files.</li>
			</ol>
			</li>
		</ol>
		</li>
	</ol>
	</li>
</ol>
6. Timestamp Participant Videos for Data Analysis
<ol>
	<li>Using the AFEA software, manually review each participant&#8217;s video and identify post-consumption time zero for each sample. Record the timestamp in a data spreadsheet. Post-consumption is defined when the sample cup is below the participant&#8217;s chin and no longer occludes the face. 
	Note: The placement of the timestamp is critical for evaluation. The point where the cup no longer occludes the face is the optimal recommendation and timestamps need to be consistent for all participants.</li>
	<li>Save the timestamp data spreadsheet (.csv) as a reference for extracting relevant data from videos. 
	Note: Participant videos may also be coded internally in the software as &#34;Event Marking&#34;.</li>
</ol>
7. Time Series Emotional Analysis
Note: Consider the &#34;baseline&#34; to be the control (i.e., unflavored water in this example). The researcher has the ability to create a different &#34;baseline treatment stimulus&#34; or a &#34;baseline time without stimulus&#34; for paired comparison dependent on the interests of the investigation. The method proposed accounts for a &#34;default&#34; state by using a paired statistical test. In other words, the procedure uses statistical blocking (i.e., a paired test) to adjust for the default appearance of each participant and therefore reduces the variability across participants.
<ol>
	<li>Extract relevant data from the exported files (.csv or .xlsx).
	<ol>
		<li>Identify a time frame relevant to the study evaluation (seconds).</li>
		<li>Manually extract respective data (time frame) from the exported participant files consulting the participant timestamp (time zero).</li>
		<li>Compile each participant&#39;s treatment data (participant number, treatment, original video time, and emotion response) per emotion (happy, neutral, sad, angry, surprised, scared, and disgusted) for the select time frame (seconds) in a new data spreadsheet for future analysis (Figure 3).</li>
		<li>Continue this process for all participants.</li>
	</ol>
	</li>
	<li>Identify the corresponding time zero from the timestamp file for each participant-treatment pair and adjust video time to a true time &#34;0&#34; for direct comparison (Figure 4, Figure 5). 
	Note: Participant data is collected in a continuous video therefore each treatment &#34;time zero&#34; is different (i.e., unflavored water video time zero is 02:13.5 and unflavored milk video time zero is 03:15.4) in Figure 4. Due to the different treatment &#34;time zeroes&#34;, the video times need to be readjusted and realigned to start at &#34;0:00.0&#34; or other standard start time in order for direct time comparison of treatment emotional response data.</li>
	<li>For each participant, emotion, and adjusted time point, extract the paired treatment (e.g., unflavored milk) and control treatment (e.g., unflavored water) quantitative emotional score. In other words, align a participant&#39;s treatment and control time series of responses for each emotion (Figure 5).</li>
	<li>Compile all participant&#39;s information (participant, adjusted time, and paired treatment (e.g., unflavored water and unflavored milk) at each time point (Figure 6). 
	Note: The steps below demonstrate the steps for a paired Wilcox test by hand. Most data analysis software programs will do this automatically. It is recommended to discuss the statistical analysis process with a statistician.</li>
	<li>Once the samples are reset and aligned with new adjusted video times, directly compare between the emotional results of a respective sample and the control (unflavored water) using sequential paired nonparametric Wilcoxon tests across the participants (Figure 7). 
	Note: The new time alignment of the samples will allow for direct comparison within the 5 seconds post-consumption time frame. If a paired observation is not present in a treatment, drop the participant from that time point comparison.
	<ol>
		<li>Calculate the difference between the control and the respective sample for each paired comparison using data spreadsheet management software. 
		Note: The comparison will be dependent on the frame rate selected for emotional analysis in the software. The protocol demonstrates 30 individual comparisons per second for 5 seconds (selected time frame). 
		Note: Use Figure 7 as a reference for columns and steps.
		<ol>
			<li>Subtract the value of milk (e.g., unflavored milk) from the value of the control (e.g., unflavored water) to determine the difference. In the data spreadsheet management software in a new column titled &#34;Treatment Difference&#34;, enter &#34;=(C2)-(D2)&#34;, where &#34;C2&#34; is the control emotional values and &#34;D2&#34; is the selected treatment emotional values. Continue this process for all time points.</li>
			<li>Calculate the absolute value of the treatment difference. In the data spreadsheet management software in a new column, enter &#34;=ABS(E2)&#34;, where &#34;E2&#34; is the Treatment Difference. Continue this process for all time points.</li>
			<li>Determine the rank order of the treatment difference. In the data spreadsheet management software in a new column, enter &#34;=RANK(G2, $G$2:$G$25, 1)&#34; where &#34;G2&#34; is the Absolute Difference and &#34;1&#34; is &#34;ascending&#34;. Continue this process for all time points.</li>
			<li>Determine the signed rank of the rank order on the spreadsheet. Change the sign to negative if the treatment difference was negative (Column I).</li>
			<li>Calculate the positive sum (=SUMIF(I2:I25, &#34;&#62;0&#34;, I2:I25) and negative sum =SUMIF(I2:I25,&#34;&#60;0&#34;,I2:I25) of the rank values.</li>
			<li>Determine the test statistic. The test statistic is the absolute value lower sum.</li>
			<li>Consult statistical tables for Wilcoxon Signed Ranked Test Statistic using the number of observations included at the specific time and a selected alpha value to determine the critical value.</li>
			<li>If the test statistic is less than the critical value reject the null hypothesis. If it is greater, accept the null hypothesis.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Graph the results on the associated treatment graph (i.e., unflavored milk compared to unflavored water) for the times when the null hypothesis is rejected. Use the sign of the difference to determine which treatment has the greater emotion (Figure 8).
	<ol>
		<li>In the data spreadsheet management software, create a graph using the values of presence or absence of significance.
		<ol>
			<li>Click &#34;Insert&#34; tab.</li>
			<li>Select &#34;Line&#34;</li>
			<li>Right click on the graph box.</li>
			<li>Click &#34;select data&#34; and follow the screen prompts to select and graph relevant data (Figure 8). 
			Note: The graphs will portray emotional results where the sample or control is higher and significant. Graph dependent, the emotion is higher at that specific time allowing the ability to discern how participant&#39;s emotions evolve over the 5 second time period between two samples. 
			Note: Statistical support with a statistician is highly recommended to extract relevant data. Development of statistical coding is required to analyze emotional results.</li>
		</ol>
		</li>
	</ol>
	</li>
</ol>

<ol>
	<li>De Wijk, R. A., Kooijman, V., Verhoeven, R. H. G., Holthuysen, N. T. E., De Graaf, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Autonomic+nervous+system+responses+on+and+facial+expressions+to+the+sight,+smell,+and+taste+of+liked+and+disliked+foods.">Autonomic nervous system responses on and facial expressions to the sight, smell, and taste of liked and disliked foods.</a> Food Qual Prefer. 26 (2), 196-203 (2012).</li><li>De Wijk, R. A., He, W., Mensink, M. G. J., Verhoeven, R. H. G., De Graaf, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=ANS+responses+and+facial+expression+differentiate+between+the+taste+of+commercial+breakfast+drinks.">ANS responses and facial expression differentiate between the taste of commercial breakfast drinks.</a> PLoS ONE. 9 (4), 1-9 (2014).</li><li>He, W., Boesveldt, S., De Graaf, C., De Wijk, R. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Behavioural+and+physiological+responses+to+two+food+odours.">Behavioural and physiological responses to two food odours.</a> Appetite. 59 (2), 628 (2012).</li><li>He, W., Boesveldt, S., De Graaf, C., De Wijk, R. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Dynamics+of+autonomic+nervous+system+responses+and+facial+expressions+to+odors.">Dynamics of autonomic nervous system responses and facial expressions to odors.</a> Front Psychol. 5 (110), 1-8 (2014).</li><li>Danner, L., Sidorkina, L., Joechl, M., Duerrschmid, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Make+a+face!+Implicit+and+explicit+measurement+of+facial+expressions+elicited+by+orange+juices+using+face+reading+technology.">Make a face! Implicit and explicit measurement of facial expressions elicited by orange juices using face reading technology.</a> Food Qual Prefer. 32 (2014), 167-172 (2013).</li><li>Danner, L., Haindl, S., Joechl, M., Duerrschmid, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Facial+expression+and+autonomous+nervous+system+responses+elicited+by+tasting+different+juices.">Facial expression and autonomous nervous system responses elicited by tasting different juices.</a> Food Res Int. 64 (2014), 81-90 (2014).</li><li>Arnade, E. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Measuring consumer emotional response to tastes and foods through facial expression analysis [thesis]. , 1-187 (2013).</li><li>Leitch, K. A., Duncan, S. E., O'Keefe, S., Rudd, R., Gallagher, D. L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Characterizing+consumer+emotional+response+to+sweeteners+using+an+emotion+terminology+questionnaire+and+facial+expression+analysis.">Characterizing consumer emotional response to sweeteners using an emotion terminology questionnaire and facial expression analysis.</a> Food Res Int. 76, 283-292 (2015).</li><li>Crist, C. A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Application+of+emotional+facial+analysis+technology+to+consumer+acceptability+using+a+basic+tastes+model.">Application of emotional facial analysis technology to consumer acceptability using a basic tastes model.</a> , (2014).</li><li>Garcia-Burgos, D., Zamora, M. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Facial+affective+reactions+to+bitter-tasting+foods+and+body+mass+index+in+adults.">Facial affective reactions to bitter-tasting foods and body mass index in adults.</a> Appetite. 71 (2013), 178-186 (2013).</li><li>Garcia-Burgos, D., Zamora, M. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Exploring+the+hedonic+and+incentive+properties+in+preferences+for+bitter+foods+via+self-reports,+facial+expressions+and+instrumental+behaviours.">Exploring the hedonic and incentive properties in preferences for bitter foods via self-reports, facial expressions and instrumental behaviours.</a> Food Qual Prefer. 39 (2015), 73-81 (2015).</li><li>Lewinski, P., Fransen, M. L., Tan, E. S. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Predicting+advertising+effectiveness+by+facial+expressions+in+response+to+amusing+persuasive+stimuli.">Predicting advertising effectiveness by facial expressions in response to amusing persuasive stimuli.</a> J. Neurosci. Psychol. Econ. 7 (1), 1-14 (2014).</li><li>Ekman, P., Friesen, W. V. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Facial+action+coding+system:+A+technique+for+the+measurement+of+facial+movement.">Facial action coding system: A technique for the measurement of facial movement.</a> , (1978).</li><li>Viola, P., Jones, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Rapid+object+detection+using+a+boosted+cascade+of+simple+features.">Rapid object detection using a boosted cascade of simple features.</a> Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern. 1, 511-518 (2001).</li><li>Sung, K. K., Poggio, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Example-based+learning+for+view-based+human+face+detection.">Example-based learning for view-based human face detection.</a> IEEE Trans. Pattern Anal. Mach. Intell. 20 (1), 39-51 (1998).</li><li>. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> FaceReader 5&#8482; Technical Specifications. , (2014).</li><li>Cootes, T., Taylor, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Statistical models of appearance for computer vision: Technical report. , (2000).</li><li>Bishop, C. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Neural networks for pattern recognition. , (1995).</li><li>Lewinski, P., den Uyl, T. M., Butler, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automated+facial+coding:+validation+of+basic+emotions+and+FACS+AUs+in+FaceReader.">Automated facial coding: validation of basic emotions and FACS AUs in FaceReader.</a> J. Neurosci. Psychol. Econ. 7 (4), 227-236 (2014).</li><li>Noldus Information Technology. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> FaceReader Reference Manual Version 6. , (2014).</li><li>Alves, N. T., Fukusima, S. S., Aznar-Casanova, J. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Models+of+brain+asymmetry+in+emotional+processing.">Models of brain asymmetry in emotional processing.</a> Psychol Neurosci. 1 (1), 63-66 (2008).</li><li>Costello, M., Clark, S., Clark, S., Costello, M., Drake, M., Bodyfelt, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Preparation+of+samples+for+instructing+students+and+staff+in+dairy+products+evaluation+(Appendix+F).">Preparation of samples for instructing students and staff in dairy products evaluation (Appendix F).</a> The sensory evaluation of dairy foods. , 551-560 (2009).</li><li>Porcherot, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=How+do+you+feel+when+you+smell+this?+Optimization+of+a+verbal+measurement+of+odor-elicited+emotions.">How do you feel when you smell this? Optimization of a verbal measurement of odor-elicited emotions.</a> Food Qual Prefer. 21, 938-947 (2010).</li><li>Warrenburg, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Effects+of+fragrance+on+emotions:+Moods+and+physiology.">Effects of fragrance on emotions: Moods and physiology.</a> Chem. Senses. 30, i248-i249 (2005).</li><li>Bredie, W. L. P., Tan, H. S. G., Wendin, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+comparative+study+on+facially+expressed+emotions+in+response+to+basic+tastes.">A comparative study on facially expressed emotions in response to basic tastes.</a> Chem. Percept. 7 (1), 1-9 (2014).</li><li>Wendin, K., Allesen-Holm, B. H., Bredie, L. P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Do+facial+reactions+add+new+dimensions+to+measuring+sensory+responses+to+basic+tastes?.">Do facial reactions add new dimensions to measuring sensory responses to basic tastes?.</a> Food Qual Prefer. 22, 346-354 (2011).</li><li>Rosenstein, D., Oster, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Differential+facial+responses+to+four+basic+tastes+in+newborns.">Differential facial responses to four basic tastes in newborns.</a> Child Dev. 59 (6), 1555-1568 (1988).</li><li>Rosenstein, D., Oster, H., P, E. k. m. a. n., E, R. o. s. e. n. b. e. r. g. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Differential+facial+responses+to+four+basic+tastes+in+newborns.">Differential facial responses to four basic tastes in newborns.</a> What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS). , 302-327 (1997).</li><li>Rozin, P., Fallon, A. E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+perspective+on+disgust.">A perspective on disgust.</a> Psychol. Rev. 94 (1), 23-41 (1987).</li><li>Delarue, J., Blumenthal, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Temporal+aspects+of+consumer+preferences.">Temporal aspects of consumer preferences.</a> Curr. Opin. Food Sci. 3, 41-46 (2015).</li><li>Sudre, J., Pineau, N., Loret, C., Marin, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Comparison+of+methods+to+monitor+liking+of+food+during+consumption.">Comparison of methods to monitor liking of food during consumption.</a> Food Qual Prefer. 24 (1), 179-189 (2012).</li><li>Weiland, R., Ellgring, H., Macht, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Gustofacial+and+olfactofacial+responses+in+human+adults.">Gustofacial and olfactofacial responses in human adults.</a> Chem. Senses. 35 (9), 841-853 (2010).</li><li>Ekman, P., Cole, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Universal+and+cultural+differences+in+facial+expressions+of+emotion.">Universal and cultural differences in facial expressions of emotion.</a> Nebraska symposium on motivation. , 207-283 (1971).</li><li>Griemel, E., Macht, M., Krumhuber, E., Ellgring, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Facial+and+affective+reactions+to+tastes+and+their+modulation+by+sadness+and+joy.">Facial and affective reactions to tastes and their modulation by sadness and joy.</a> Physiol Behav. 89 (2), 261-269 (2006).</li></ol>

The authors have nothing to disclose.

AFEA application in literature related to food and beverage is very limited1-11. The application to food is new, creating an opportunity for establishing methodology and data interpretation. Arnade (2013)7 found high individual variability among individual emotional response to chocolate milk and white milk using area under the curve analysis and analysis of variance. However, even with participant variability, participants generated a happy response longer while sad and disgusted had shorter time response7. In a separate study using high and low concentrations of basic tastes, Arnade (2013)7, found that the differences in emotional response among basic tastes as well as between two levels of basic taste intensities (high and low intensity), were not as significant as expected, thereby questioning the accuracy of current AFEA methodology and data analysis. Sensory evaluation of foods and beverages is a complex and dynamic response process30. Temporal changes can occur throughout oral processing and swallowing thus potentially influencing the acceptability of the stimuli over time30. For this reason, it may beneficial to measure evaluator response throughout the entire eating experience. Specific oral processing times have been suggested (initial contact with tongue, mastication, swallowing, etc.)31, but none are standardized and times are largely dependent on the project and the researcher&#39;s discretion30.
The proposed emotional time series analysis was able to detect emotional changes and statistical differences between the control (unflavored water) and respective treatments. Moreover, emotional profiles associated with acceptability may aid in anticipating behavior related to foods and beverages. Results show that distinguishable time series trends exist with AFEA related to flavors in milk (Figures 10, 11, and 12). The time series analysis assists in differentiating food acceptability across a population by integrating characterized emotions (Figure 10, 11, and 12) as well as supporting hedonic acceptability trends (Figure 9). Leitch et al.8 observed differences between sweeteners and the water baseline using time series analysis (5 sec), and also found that the utilization of time series graphs provided for better interpretation of data and results. Moreover, emotional changes can be observed over time and emotional response treatment differences may be determined at different time points or intervals. For example, Leitch et al.8 observed that the approach emotions (angry, happy and surprised) were observed between the artificial sweetener-water comparisons but were observed at different times over the 5 sec observation window. However, Leitch et al.8 did not establish directionality of expression, making it difficult to understand the emotional difference between the control (water) and the treatment (unsweetened tea) using their graphical interpretation and presentation. The modified and improved time series analysis methodology presented in our study allows for statistical difference directionality. The directionality and results plotting allows researchers to visualize where statistically relevant emotional changes occur over the selected time frame.
Reducing video analysis failures is essential for attaining valid data and effectively using time and personnel resources. Critical steps and troubleshooting steps in the protocol include optimizing the participant sensory environment (lighting, video camera angle, chair height, thorough participant guidance instructions, etc.). Also, participants should be screened and excluded if they fall into a software incompatibility category (i.e., thick framed glasses, heavily bearded faces and skin tone) (Figure 2). These factors will influence AFEA fit modeling, emotional categorization, and data output. If a significant portion of a participant&#39;s data output consists of &#34;FIT_FAILED&#34; and &#34;FIND_FAILED&#34;, data should be reevaluated for inclusion in the time series analysis (Figure 1). Time series analysis cannot be utilized if data output log files predominantly contain &#34;FIT_FAILED&#34; and &#34;FIND_FAILED&#34; as this is bad data (Figure 1). Shadowing on the face due to lighting settings may severely inhibit video capture quality, resulting in poor video collection. To avoid intense shadowing, diffuse frontal lighting is ideal while the light intensity or color is not as relevant20. Intense overhead lighting should be reduced as it can promote shadows on the face20. A dark background behind the participant is recommended20. It is suggested from the AFEA software manufacturer to place the setup in front of a window to have diffuse daylight lighting20. Also, if using a computer monitor, two lights may be placed on either side of the user&#39;s face for illumination and shadow reduction20. Additionally, professional photo lights may be used to counteract undesirable environment lighting20. Ultimately, it is up to the discretion of the researcher, individual protocol/methodology, and environment to control lighting for capture. It is recommended to discuss the data capture environment and the tools with the software provider before purchase and installation. Furthermore, chair height and camera angle are important to adjust individually for each participant. The participant should be comfortable but at a height where the camera is straight on the face. An attempt to reduce the camera angle on the face is encouraged for optimizing the AFEA video capture. Lastly, it is imperative to give verbal instructions to the participants prior to sampling. Participant behavior during video capture may limit data collection due to facial occlusion, movements, and camera avoidance.
For participant sample size needed for a study, the authors recommend a range of 10 to 50 participants. Although a small number will provide almost no statistical power, at least 2 participants are needed in general for time series analysis. Participant variability is high, and in the early stages of this research there is no guidance to offer with sample size. Sample size will vary depending on flavors, flavor intensity, and expected treatment acceptability. Samples with smaller flavor differences will require more participants. The 30 second controlled sampling period encompasses a time span adequate for the entire sampling evaluation period (i.e., showing the index card, opening a sample (removing the lid), consumption, and emotional capture). The entire 30 seconds is not used in data analysis. The benefit of this designated 30 second capture time is that the researcher can decide the pertinent evaluation time to be used in data analysis. The 30 second time window can assist in selecting a time frame of interest during a video sample while coding or timestamping videos. Ultimately, the time window is up to the discretion of the researcher. In our example, we used the 5 sec sampling window post-consumption. Furthermore, the present methodology defines time zero when the sample cup no longer occludes the face (cup at the chin). It is critically important to lessen the time between consumption and sample cup facial occlusion due to brief and changing emotions. Due to sample cup facial occlusion the initial time where the sample makes contact with the tongue is unreliable data (see Figure 1). Therefore, the point where the cup no longer occludes the face is the optimal recommendation. Timestamps need to be consistent for all participants. The color card is a convenient way for researchers to identify treatments in the video and mark the appropriate time frame (time zero) for sample evaluation. The color cards are especially helpful if treatments are in random order and serve as an extra validation of sample identification in the continuous video.
Limitations of this technique exist as participants may not follow directions or unavoidable shadowing on the participant&#39;s face may cause face fit model failures (Figure 2). However, the suggested critical steps offer ways to mitigate and reduce these interferences. Additionally, time series analysis will not read exported log files with files predominantly containing &#34;FIT_FAILED&#34; and &#34;FIND_FAILED&#34; (Figure 1). These file cannot be salvaged and will not be able to be included in time series analysis. Also, the consumption of food and beverages still may alter the facial structure in such a way to distort the emotional categorization. Hard or chewy foods require extensive jaw motion. Use of a drinking straw and associated sucking, also causes facial occlusion (straw) and distorts the face (sucking). This observation is based on preliminary data from our laboratory research. The software facial model cannot discern the differences between chewing (or sucking) and motor expressions associated with emotional categorization. With food and beverage samples, the opportunity for facial occlusion is higher than that of viewing videos and pictures. Participants must bring the sample to the face and remove the container from the face thus interrupting the software model and potentially reducing valuable emotional information (See Figure 1). As mentioned previously, emotions happen quickly and for a short duration. It is important to reduce the facial occlusion in an effort to capture emotions. The proposed methodology makes treatment comparisons at one thirtieth of a second to find changes in emotional patterns and changes in emotional duration across time. With the proposed methodology, patterns of emotional longevity are important. Unfortunately, emotional categorization problems can occur. Most notably there is a problem categorizing happy and disgust6, 9, 32, 33, 34. Oftentimes, this is due to participants masking their distaste or surprised feeling by smiling6, 32, 33, 34 that could be due to a &#34;social display rule&#34;32. Furthermore, the AFEA software is limited to seven emotional categories (neutral, happy, sad, scared, surprised, angry and disgusted). Emotional response to foods and beverages may be more complex than the current AFEA classification of universal emotions and categorization may be different in response to a food or beverage stimuli. Manual coding using FACS has been applied to gustofacial and olfactofacial responses of basic tastes and an assortment of odors and appeared to be sensitive enough to detect treatment differences in regards to AUs32. FACS is tedious and very time consuming, however, the temporal application of absence or presence of AUs may be useful to assist with complex responses that AFEA might not classify correctly or if emotional results are unexpected. While time series data allows for facial classifications to occur simultaneously and with significant expression, caution should be used with translating results into a single emotion due to emotional complexity.
The proposed methodology and data analysis technique may be applied to other beverages and soft foods. AFEA software was able to identify emotions to flavored and unflavored samples. The proposed methodology and temporal analysis may aid with characterizing implicit responses thereby providing new advances in emotional responses and behaviors of a population relating to food. Future applications of this technique may expand into other beverage categories or soft foods. We have demonstrated methodology to attain video capture for emotional response and data analysis methodology. We aim to create a standard approach for both emotional AFEA capture and emotional time series analysis. The method approach has shown success in our research. We hope to expand and apply this approach for evaluating emotional response to foods and beverages and the relationship to choice and behaviors.

Automated facial expression analysis (AFEA) is a prospective analytical tool for characterizing emotional responses to beverages and foods. Emotional analysis can add an extra dimension to existing sensory science methodologies, food evaluation practices, and hedonic scale ratings typically used both in research and industry settings. Emotional analysis could provide an additional metric that reveals a more accurate response to foods and beverages. Hedonic scoring may include participant bias due to failure to record reactions1.
AFEA research has been used in many research applications including computer gaming, user behavior, education/pedagogy, and psychology studies on empathy and deceit. Most food-associated research has focused on characterizing emotional response to food quality and human behavior with food. With the recent trend in gaining insights into food behaviors, a growing body of literature reports use of AFEA for characterizing the human emotional response associated with foods, beverages, and odorants1-12.
AFEA is derived from the Facial Action Coding System (FACS). The facial action coding system (FACS) discriminates facial movements characterized by action units (AUs) on a 5-point intensity scale13. The FACS approach requires trained review experts, manual coding, significant evaluation time, and provides limited data analysis options. AFEA was developed as a rapid evaluation method to determine emotions. AFEA software relies on facial muscular movement, facial databases, and algorithms to characterize the emotional response14-18. The AFEA software used in this study reached a &#34;FACS index of agreement of 0.67 on average on both the Warsaw Set of Emotional Facial Expression Pictures (WSEFEP) and Amsterdam Dynamic Facial Expression Set (ADFES), which is close to a standard agreement of 0.70 for manual coding&#34;19. Universal emotions included in the analysis are happy (positive), sad (negative), disgusted (negative), surprised (positive or negative), angry (negative), scared (negative) and neutral each on a separate scale of 0 to 1 (0=not expressed; 1=fully expressed)20. In addition, psychology literature includes happy, surprised, and angry as &#34;approach&#34; emotions (toward stimuli) and sad, scared, and disgusted as &#34;withdrawal&#34; emotions (away from aversive stimuli)21.
One limitation of the current AFEA software for characterizing emotions associated with foods is interference from facial movements associated with chewing and swallowing as well as other gross motor motions, such as extreme head movements. The software targets smaller facial muscular motions, relating position and degree of movement, based on over 500 muscle points on the face16,17. Chewing motions interfere with classification of expressions. This limitation may be addressed using liquefied foods. However, other methodology challenges can also decrease video sensitivity and AFEA analysis including data collection environment, technology, researcher instructions, participant behavior, and participant attributes.
A standard methodology has not been developed and verified for optimal video capture and data analysis using AFEA for emotional response to foods and beverages in a sensory evaluation laboratory setting. Many aspects can affect the video capture environment including lighting, shadowing due to lighting, participant directions, participant behavior, participant height, as well as, camera height, camera angling, and equipment settings. Moreover, data analysis methodologies are inconsistent and lack a standard methodology for assessing emotional response. Here, we will demonstrate our standard operating procedure for capturing emotional data and processing data into meaningful results using beverages (flavored milk, unflavored milk and unflavored water) for evaluation. To our knowledge only one peer reviewed publication, from our lab group, has utilized time series for data interpretation for emotions analysis8; however, the method has been updated for our presented method. Our aim is to develop an improved and consistent methodology to help with reproducibility in a sensory evaluation laboratory setting. For demonstration, the objective of the study model is to evaluate if AFEA could supplement traditional hedonic acceptability assessment of flavored milk, unflavored milk and unflavored water. The intention of this video protocol is to help establish AFEA methodology, standardize video capture criteria in a sensory evaluation laboratory (sensory booth setting), and illustrate a method for temporal emotional data analysis of a population.

<table><tbody><tr><td>2% Reduced Fat Milk</td><td>Kroger Brand, Cincinnati, OH or DZA Brands, LLC, Salisbury, NC</td><td>na</td><td>for solutions</td></tr><tr><td>Drinking Water</td><td>Kroger Brand, Cincinnati, OH</td><td>na</td><td>for solutions</td></tr><tr><td>Imitation Clear Vanilla Flavor</td><td>Kroger Brand, Cincinnati, OH</td><td>na</td><td>for solutions</td></tr><tr><td>Iodized Salt</td><td>Kroger Brand, Cincinnati, OH</td><td>na</td><td>for solutions</td></tr><tr><td>FaceReader 6</td><td>Noldus Information Technology, Wageningen, The Netherlands</td><td>na</td><td>For Facial Analysis</td></tr><tr><td>Sensory Information Management System (SIMS) 2000</td><td>Sensory Computer Systems, Berkeley Heights, NJ</td><td>Version 6</td><td>For Sensory Data Capture</td></tr><tr><td>Rhapsody</td><td>Acuity Brands Lighting, Inc., Conyers, GA</td><td></td><td>For Environment Illumination</td></tr><tr><td>R Version&amp;nbsp;</td><td>R Core Team 2015</td><td>3.1.1</td><td>For Statistical Analysis</td></tr><tr><td>Microsoft Office</td><td>Microsoft</td><td>na</td><td>For Statistical Analysis</td></tr><tr><td>JMP</td><td>Statistical Analysis Software (SAS) Version 9.2, SAS Institute, Cary, NC</td><td>na</td><td>For Statistical Analysis</td></tr><tr><td>Media Recorder 2.5</td><td>Noldus Information Technology, Wageningen, The Netherlands</td><td>na</td><td>For capturing participants sensory evaluation</td></tr><tr><td>Axis M1054 Camera</td><td>Axis Communications, Lund, Sweden</td><td>na</td><td></td></tr><tr><td>Beverage</td><td></td><td>na</td><td>Beverage or soft food for evaluation</td></tr></tbody></table>

protocol for data collection and analysis applied to automated facial expression analysis technology and temporal analysis for sensory evaluation

We demonstrate a method for capturing emotional response to beverages and liquefied foods in a sensory evaluation laboratory using automated facial expression analysis (AFEA) software. Additionally, we demonstrate a method for extracting relevant emotional data output and plotting the emotional response of a population over a specified time frame. By time pairing each participant's treatment response to a control stimulus (baseline), the overall emotional response over time and across multiple participants can be quantified. AFEA is a prospective analytical tool for assessing unbiased response to food and beverages. At present, most research has mainly focused on beverages. Methodologies and analyses have not yet been standardized for the application of AFEA to beverages and foods; however, a consistent standard methodology is needed. Optimizing video capture procedures and resulting video quality aids in a successful collection of emotional response to foods. Furthermore, the methodology of data analysis is novel for extracting the pertinent data relevant to the emotional response. The combinations of video capture optimization and data analysis will aid in standardizing the protocol for automated facial expression analysis and interpretation of emotional response data.

The method proposes a standard protocol for AFEA data collection. If suggested protocol steps are followed, unusable emotional data output (Figure 1) resulting from poor data collection (Figure 2: A; Left Picture) may be limited. Time series analysis cannot be utilized if log files (.txt) predominantly contain &#34;FIT_FAILED&#34; and &#34;FIND_FAILED&#34; as this is bad data (Figure 1). Furthermore, the method includes a protocol for direct statistical comparison between two treatments of emotional data output over a time frame to establish an emotional profile. Time series analysis can provide emotional trends over time and can provide a value-added dimension to hedonic acceptability results. Additionally, time series analysis can show changes in emotional levels over time, which is valuable during the eating experience.
Unflavored milk, unflavored water and vanilla extract flavor in milk were not different (p&#62;0.05) in mean acceptability scores and were rated as &#34;liked slightly&#34; (Figure 9). Hedonic results infer that there were not any acceptability differences between unflavored milk, unflavored water and vanilla extract flavor in milk. However, AFEA time series analysis indicated unflavored milk generated less disgusted (p&#60;0.025; 0 sec), surprised (p&#60;0.025; 0-2.0 sec), less sad (p&#60;0.025; 2.0-2.5 sec) and less neutral (p&#60;0.025; ~3.0-3.5 sec) responses than did unflavored water (Figure 10). Additionally, vanilla extract flavor in milk introduced more happy expressions just before 5.0 seconds (p&#60;0.025) and less sad (p&#60;0.025; 2.0-3.0 and 5.0 sec) than unflavored water (Figure 11). Vanilla, as an odor, has been associated with the terms &#34;relaxed&#34;, &#34;serene&#34;, &#34;reassured&#34;, &#34;happiness&#34;, &#34;well-being&#34;, &#34;pleasantly surprised&#34;23 and &#34;pleasant&#34;24. Salty flavor in milk had lower (p&#60;0.05) mean hedonic acceptability scores (disliked moderately) (Figure 9) and salty flavor in milk generated more disgust (p&#60;0.025) later (3.0-5.0 sec) than unflavored water (Figure 12). Intense salty has been associated with disgust and surprised25, 26. However, some studies have stated that salty flavor does not elicit facial response7, 27-29.
<img alt="Figure 1" src="/files/ftp_upload/54046/54046fig1.jpg" /> 
Figure 1. Example of sub-optimal data capture due to participant incompatibility with AFEA software resulting in loss of raw emotional data response points in the exported output files [FIT_FAILED; FIND_FAILED]. Video failures occur when serious facial occlusions or the inability to map the face persists during the specified post-consumption window. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig1large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" src="/files/ftp_upload/54046/54046fig2.jpg" /> 
Figure 2. Example of sub-optimal data capture due to participant software modeling. The figure presents sub-optimal data capture due to participant software modeling incompatibility and failure of face mapping to determine emotional response (A). Example of successful fit modeling and ability to capture participant&#39;s emotional response (B). <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" src="/files/ftp_upload/54046/54046fig3.jpg" /> 
Figure 3. Example of extracted participant data compiled in a new data spreadsheet. Participant data (participant number, treatment, original video time, and emotion response) is identified per emotion (happy, neutral, sad, angry, surprised, scared, and disgusted) for the select time frame (seconds). This spreadsheet is utilized for subsequent analyses. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig3large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" src="/files/ftp_upload/54046/54046fig4.jpg" /> 
Figure 4. Example of extracted participant data compiled for subsequent analysis. The extracted participant data (A1 and B1) is compiled (A2 and B2), graphed (A3 and B3) and aligned (A4 and B4) as a visual for direct comparison. The respective time zero for control (A4: Surprised Unflavored Water) and treatment (B4: Surprised Unflavored Milk) are displayed for comparing the surprised emotional results. This example represents and identifies the corresponding time zero from the timestamp file for each participant-treatment pair. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig4large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 5" src="/files/ftp_upload/54046/54046fig5.jpg" /> 
Figure 5. Example of extracted participant data with adjusted time frame. The extracted participant data is presented with adjusted time frame with a true &#34;time zero&#34; (A1 and B1). The time adjustment allows for direct comparison between a control (A: Surprised Unflavored Water) and a treatment (B2: Surprised Unflavored Milk) (A2 and B2). This example represents and identifies the corresponding true &#34;time zero&#34; (adjusted) from the timestamp file for each participant-treatment pair. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig5large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 6" src="/files/ftp_upload/54046/54046fig6.jpg" /> 
Figure 6. Example of the process for compiling all participants&#39; data. The participant, adjusted time, and paired treatment (e.g., unflavored water and unflavored milk) at each time point is compiled to prepare for statistical analysis. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig6large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 7" src="/files/ftp_upload/54046/54046fig7.jpg" /> 
Figure 7. Data spreadsheet example comparing a control (Unflavored Water) and a treatment (Unflavored Milk) using Wilcoxon tests across participants at a specific time point. The figure represents direct comparison between the emotional results of a respective sample and the control (unflavored water) using sequential paired nonparametric Wilcoxon tests across the participants. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig7large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 8" src="/files/ftp_upload/54046/54046fig8.jpg" /> 
Figure 8. Example of the data spreadsheet to graph the results if (p&#60;0.025) on the associated treatment graph (i.e., unflavored milk compared to unflavored water). Results of sequential paired nonparametric Wilcoxon tests across the participants are graphed for the times where the null hypothesis is rejected. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig8large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 9" src="/files/ftp_upload/54046/54046fig9.jpg" /> 
Figure 9. Mean acceptability (hedonic) scores of unflavored water, unflavored milk, vanilla extract flavor in milk and salty flavor in milk beverage solutions. Acceptability was based on a 9-point hedonic scale (1=dislike extremely, 5=neither like nor dislike, 9=like extremely; mean +/- SD)1. Treatment means with different superscripts significantly differ in liking (p&#60;0.05). Unflavored milk, unflavored water and vanilla extract flavor in milk were not different (p&#62;0.05) in mean acceptability scores and were rated as &#34;liked slightly&#34;. Salty flavor in milk had a lower (p&#60;0.05) mean acceptability scores (disliked moderately). <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig9large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 10" src="/files/ftp_upload/54046/54046fig10.jpg" /> 
Figure 10. Time series graphs of classified emotions on automated facial expression analysis data over 5.0 seconds comparing unflavored milk and unflavored water. Based on sequential paired nonparametric Wilcoxon tests between unflavored milk and unflavored water (baseline), results are plotted on the respective treatment graph if the treatment median is higher and of greater significance (p&#60;0.025) for each emotion. Presence of a line indicates a significant difference (p&#60;0.025) at the specific time point where the median is higher, while absence of a line indicates no difference at a specific time point (p&#62;0.025). Absence of lines in unflavored milk (A) reveals no emotional categorization compared to unflavored water (p&#60;0.025) over 5.0 seconds. In the unflavored water (B), emotional results compared to unflavored milk reveal disgusted (crimson line) at 0 sec, surprised (orange line) occurs between 0 - 1.5 sec, sad (green line) occurs around 2.5 sec, and neutral (red line) occurs around 3 - 3.5 sec (p&#60;0.025). <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig10large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 11" src="/files/ftp_upload/54046/54046fig11.jpg" /> 
Figure 11. Time series graphs of classified emotions based on automated facial expression analysis data over 5.0 seconds comparing vanilla extract flavor in milk and unflavored water (baseline). Based on sequential paired nonparametric Wilcoxon tests between vanilla extract flavor in milk and unflavored water, results are plotted on the respective treatment graph if treatment median is higher and of greater significance (p&#60;0.025) for each emotion. Presence of a line indicates a significant difference (p&#60;0.025) at the specific time point where the median is higher, while absence of a line indicates no difference at a specific time point (p&#62;0.025). Vanilla extract flavor in milk (A) shows happy just before 5 sec (blue line) while unflavored water (B) displays more sad around 2 - 2.5 and 5 sec (green line) (p&#60;0.025). <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig11large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 12" src="/files/ftp_upload/54046/54046fig12.jpg" /> 
Figure 12. Time series graphs of classified emotions based on automated facial expression analysis data over 5.0 seconds comparing salty flavor in milk and unflavored water. Based on sequential paired nonparametric Wilcoxon tests between salty flavor in milk and unflavored water (baseline), results are plotted on the respective treatment graph if treatment median is higher and of greater significance (p&#60;0.025) for each emotion. Presence of a line indicates a significant difference (p&#60;0.025) at the specific time point where the median is higher, while absence of a line indicates no difference at a specific time point (p&#62;0.025). Salty flavor in milk (A) has significant disgust from 3 - 5 seconds (crimson line) while unflavored water (B) has disgust at the beginning (crimson line) and more neutral from 2 - 5 seconds (red line) (p&#60;0.025). <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/54046/54046fig12large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

Watch this Scientific Journal Video about Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation at JoVE.com

Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation

A protocol for capturing and statistically analyzing emotional response of a population to beverages and liquefied foods in a sensory evaluation laboratory using automated facial expression analysis software is described.

We demonstrate a method for capturing emotional response to beverages and liquefied foods in a sensory evaluation laboratory using automated facial ...

protocol-for-data-collection-analysis-applied-to-automated-facial

Nanyang Technological University

Research

JoVE Journal

Behavior

9.4K Views.  Virginia Tech.  Een protocol voor het verzamelen en statistisch analyseren emotionele reactie van een populatie aan dranken en vloeibare voedingsmiddelen in een sensorische evaluatie laboratorium met geautomatiseerde gezichtsuitdrukking analysesoftware wordt beschreven. 

Video: Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation

A Protocol for Comprehensive Assessment of Bulbar Dysfunction in Amyotrophic Lateral Sclerosis (ALS)

Improved methods for assessing bulbar impairment are necessary for expediting diagnosis of bulbar dysfunction in ALS, for predicting disease progression across speech subsystems, and for addressing the critical need for sensitive outcome measures for ongoing experimental treatment trials. To address this need, we are obtaining longitudinal profiles of bulbar impairment in 100 individuals based on a comprehensive instrumentation-based assessment that yield objective measures. Using instrumental approaches to quantify speech-related behaviors is very important in a field that has primarily relied on subjective, auditory-perceptual forms of speech assessment1. Our assessment protocol measures performance across all of the speech subsystems, which include respiratory, phonatory (laryngeal), resonatory (velopharyngeal), and articulatory. The articulatory subsystem is divided into the facial components (jaw and lip), and the tongue. Prior research has suggested that each speech subsystem responds differently to neurological diseases such as ALS. The current protocol is designed to test the performance of each speech subsystem as independently from other subsystems as possible. The speech subsystems are evaluated in the context of more global changes to speech performance. These speech system level variables include speaking rate and intelligibility of speech.
The protocol requires specialized instrumentation, and commercial and custom software. The respiratory, phonatory, and resonatory subsystems are evaluated using pressure-flow (aerodynamic) and acoustic methods. The articulatory subsystem is assessed using 3D motion tracking techniques. The objective measures that are used to quantify bulbar impairment have been well established in the speech literature and show sensitivity to changes in bulbar function with disease progression. The result of the assessment is a comprehensive, across-subsystem performance profile for each participant. The profile, when compared to the same measures obtained from healthy controls, is used for diagnostic purposes. Currently, we are testing the sensitivity and specificity of these measures for diagnosis of ALS and for predicting the rate of disease progression. In the long term, the more refined endophenotype of bulbar ALS derived from this work is expected to strengthen future efforts to identify the genetic loci of ALS and improve diagnostic and treatment specificity of the disease as a whole. The objective assessment that is demonstrated in this video may be used to assess a broad range of speech motor impairments, including those related to stroke, traumatic brain injury, multiple sclerosis, and Parkinson disease.

A Protocol for Comprehensive Assessment of Bulbar Dysfunction in Amyotrophic Lateral Sclerosis ALS

Objective assessments of the physiological mechanisms that support speech are needed to monitor disease onset and progression in persons with ALS and to quantify treatment effects in clinical trials. In this video, we present a comprehensive, instrumentation-based protocol for quantifying speech motor performance in clinical populations.

Improved methods for assessing bulbar impairment are necessary for expediting diagnosis of bulbar dysfunction in ALS, for predicting disease progression ...

Medicine

Experimental Assessment of Mouse Sociability Using an Automated Image Processing Approach

Mouse is the preferred model organism for testing drugs designed to increase sociability. We present a method to quantify mouse sociability in which the test mouse is placed in a standardized apparatus and relevant behaviors are assessed in three different sessions (called session I, II, and III).
The apparatus has three compartments (see Figure 1), the left and right compartments contain an inverted cup which can house a mouse (called &#34;stimulus mouse&#34;).
In session I, the test mouse is placed in the cage and its mobility is characterized by the number of transitions made between compartments. In session II, a stimulus mouse is placed under one of the inverted cups and the sociability of the test mouse is quantified by the amounts of time it spends near the cup containing the enclosed stimulus mouse vs. the empty inverted cup. In session III, the inverted cups are removed and both mice interact freely. The sociability of the test mouse in session III is quantified by the number of social approaches it makes toward the stimulus mouse and by the number of times it avoids a social approach by the stimulus mouse.
The automated evaluation of the movie detects the nose of the test mouse, which allows the determination of all described sociability measures in session I and II (in session III, approaches are identified automatically but classified manually). To find the nose, the image of an empty cage is digitally subtracted from each frame of the movie and the resulting image is binarized to identify the mouse pixels. The mouse tail is automatically removed and the two most distant points of the remaining mouse are determined; these are close to nose and base of tail. By analyzing the motion of the mouse and using continuity arguments, the nose is identified.
<img alt="Figure 1" src="/files/ftp_upload/52508/52508fig1.jpg" /> 
Figure 1. Assessment of Sociability During 3 sessions. Session I (top): Acclimation of test mouse to the cage. Session II (middle): Test mouse moving freely in the cage while the stimulus mouse is enclosed in an inverted cup. Session III (bottom): Both test mouse and stimulus mouse are allowed to move freely and interact with each other.

This protocol describes a method to quantify mouse sociability. Mice are videotaped as they move and interact in a special cage. Movie processing allows for the automated quantification of sociability with excellent accuracy and reliability.

Mouse is the preferred model organism for testing drugs designed to increase sociability. We present a method to quantify mouse sociability in which the ...

Assessing the Multiple Dimensions of Engagement to Characterize Learning: A Neurophysiological Perspective

In a recent theoretical synthesis on the concept of engagement, Fredricks, Blumenfeld and Paris1 defined engagement by its multiple dimensions: behavioral, emotional and cognitive. They observed that individual types of engagement had not been studied in conjunction, and little information was available about interactions or synergy between the dimensions; consequently, more studies would contribute to creating finely tuned teaching interventions. Benefiting from the recent technological advances in neurosciences, this paper presents a recently developed methodology to gather and synchronize data on multidimensional engagement during learning tasks. The technique involves the collection of (a) electroencephalography, (b) electrodermal, (c) eye-tracking, and (d) facial emotion recognition data on four different computers. This led to synchronization issues for data collected from multiple sources. Post synchronization in specialized integration software gives researchers a better understanding of the dynamics between the multiple dimensions of engagement. For curriculum developers, these data could provide informed guidelines for achieving better instruction/learning efficiency. This technique also opens up possibilities in the field of brain-computer interactions, where adaptive learning or assessment environments could be developed.

This paper aims to describe the techniques involved in the collection and synchronization of the multiple dimensions (behavioral, affective and cognitive) of learners&#8217; engagement during a task.

In a recent theoretical synthesis on the concept of engagement, Fredricks, Blumenfeld and Paris1 defined engagement by its multiple dimensions: ...

Exploring the Use of Isolated Expressions and Film Clips to Evaluate Emotion Recognition by People with Traumatic Brain Injury

The current study presented 60 people with traumatic brain injury (TBI) and 60 controls with isolated facial emotion expressions, isolated vocal emotion expressions, and multimodal (i.e., film clips) stimuli that included contextual cues. All stimuli were presented via computer. Participants were required to indicate how the person in each stimulus was feeling using a forced-choice format. Additionally, for the film clips, participants had to indicate how they felt in response to the stimulus, and the level of intensity with which they experienced that emotion.

This paper describes how to implement a battery of behavioral tasks to examine emotion recognition of isolated facial and vocal emotion expressions, and a novel task using commercial television and film clips to assess multimodal emotion recognition that includes contextual cues.

The current study presented 60 people with traumatic brain injury (TBI) and 60 controls with isolated facial emotion expressions, isolated vocal emotion ...

Conscious and Non-conscious Representations of Emotional Faces in Asperger's Syndrome

Several neuroimaging studies have suggested that the low spatial frequency content in an emotional face mainly activates the amygdala, pulvinar, and superior colliculus especially with fearful faces1-3. These regions constitute the limbic structure in non-conscious perception of emotions and modulate cortical activity either directly or indirectly2. In contrast, the conscious representation of emotions is more pronounced in the anterior cingulate, prefrontal cortex, and somatosensory cortex for directing voluntary attention to details in faces3,4. Asperger&#39;s syndrome (AS)5,6 represents an atypical mental disturbance that affects sensory, affective and communicative abilities, without interfering with normal linguistic skills and intellectual ability. Several studies have found that functional deficits in the neural circuitry important for facial emotion recognition can partly explain social communication failure in patients with AS7-9. In order to clarify the interplay between conscious and non-conscious representations of emotional faces in AS, an EEG experimental protocol is designed with two tasks involving emotionality evaluation of either photograph or line-drawing faces. A pilot study is introduced for selecting face stimuli that minimize the differences in reaction times and scores assigned to facial emotions between the pretested patients with AS and IQ/gender-matched healthy controls. Information from the pretested patients was used to develop the scoring system used for the emotionality evaluation. Research into facial emotions and visual stimuli with different spatial frequency contents has reached discrepant findings depending on the demographic characteristics of participants and task demands2. The experimental protocol is intended to clarify deficits in patients with AS in processing emotional faces when compared with healthy controls by controlling for factors unrelated to recognition of facial emotions, such as task difficulty, IQ and gender.

An EEG experimental protocol is designed to clarify the interplay between conscious and non-conscious representations of emotional faces in patients with Asperger&#39;s syndrome. The technique suggests that patients with Asperger&#39;s syndrome have deficits in non-conscious representation of emotional faces, but have comparable performance in conscious representation with healthy controls.

Several neuroimaging studies have suggested that the low spatial frequency content in an emotional face mainly activates the amygdala, pulvinar, and ...

Using Facial Electromyography to Assess Facial Muscle Reactions to Experienced and Observed Affective Touch in Humans

"Affective" touch is believed to be processed in a manner distinct from discriminatory touch and to involve activation of C-tactile (CT) afferent fibers. Touch that optimally activates CT fibers is consistently rated as hedonically pleasant. Patient groups with impaired social-emotional functioning also show disordered affective touch ratings. However, relying on self-reported ratings of touch has many limitations, including recall bias and communication barriers. Here, we describe a methodological approach to study affective responses to touch via facial electromyography (EMG) that circumvents the reliance on self-report ratings. Facial EMG is an objective, quantitative, and non-invasive method to measure facial muscle activity indicative of affective responses. Responses can be assessed across healthy and patient populations without the need for verbal communication. Here, we provide two separate datasets demonstrating that CT-optimal and non-optimal touch elicit distinct facial muscle reactions. Moreover, facial EMG responses are consistent across stimulus modalities, e.g. tactile (experienced touch) and visual (observed touch). Finally, the temporal resolution of facial EMG can detect responses on timescales that supersede that of verbal reporting. Together, our data suggest that facial EMG is a suitable methodology for use in affective tactile research that can be used to supplement, or in some cases, supplant, existing measures.

We describe a protocol to assess facial muscle activity in response to experienced and observed tactile stimulation using facial electromyography.

"Affective" touch is believed to be processed in a manner distinct from discriminatory touch and to involve activation of C-tactile (CT) afferent fibers. ...

Dynamic Digital Biomarkers of Motor and Cognitive Function in Parkinson's Disease

As Parkinson&#8217;s disease (PD) is a heterogeneous disorder, personalized medicine is truly required to optimize care. In their current form, standard scores from paper and pencil symptom- measures traditionally used to track disease progression are too coarse (discrete) to capture the granularity of the clinical phenomena under consideration, in the face of tremendous symptom diversity. For this reason, sensors, wearables, and mobile devices are increasingly incorporated into PD research and routine care. These digital measures, while more precise, yield data that are less standardized and interpretable than traditional measures, and consequently, the two types of data remain largely siloed. Both of these issues present barriers to the broad clinical application of the field&#8217;s most precise assessment tools. This protocol addresses both problems. Using traditional tasks to measure cognition and motor control, we test the participant, while co-registering biophysical signals unobtrusively using wearables. We then integrate the scores from traditional paper-and-pencil methods with the digital data that we continuously register. We offer a new standardized data type and unifying statistical platform that enables the dynamic tracking of change in the person&#8217;s stochastic signatures under different conditions that probe different functional levels of neuromotor control, ranging from voluntary to autonomic. The protocol and standardized statistical framework offer dynamic digital biomarkers of physical and cognitive function in PD that correspond to validated clinical scales, while significantly improving their precision.&#160;

This protocol offers a digitization of portions of traditional clinical tasks commonly used to measure cognition and motor control in Parkinson&#8217;s disease. Clinical tasks are digitized while biophysical rhythms are co-registered from different functional levels of the nervous systems, ranging from voluntary, spontaneous, automatic to autonomic.

As Parkinson&#8217;s disease (PD) is a heterogeneous disorder, personalized medicine is truly required to optimize care. In their current form, standard ...

Real-Time Mouse Eye Squint Detection for Migraine Pain Tracking Using DeepLabCut

An Automated Squint Method for Time-syncing Behavior and Brain Dynamics in Mouse Pain Studies

Spontaneous pain has been challenging to track in real time and quantify in a way that prevents human bias. This is especially true for metrics of head pain, as in disorders such as migraine. Eye squint has emerged as a continuous variable metric that can be measured over time and is effective for predicting pain states in such assays. This paper provides a protocol for the use of DeepLabCut (DLC) to automate and quantify eye squint (Euclidean distance between eyelids) in restrained mice with freely rotating head motions. This protocol enables unbiased quantification of eye squint to be paired with and compared directly against mechanistic measures such as neurophysiology. We provide an assessment of AI training parameters necessary for achieving success as defined by discriminating squint and non-squint periods. We demonstrate an ability to reliably track and differentiate squint in a CGRP-induced migraine-like phenotype at a sub second resolution.

Author Spotlight: Deciphering Electrical Networks Behind Complex Brain Activities and Disorders

This protocol provides a method for tracking automated eye squint in rodents over time in a manner compatible with time-locking to neurophysiological measures. This protocol is expected to be useful to researchers studying mechanisms of pain disorders such as migraine.

Spontaneous pain has been challenging to track in real time and quantify in a way that prevents human bias. This is especially true for metrics of head ...

Enhancing Classroom Engagement: AI-Based Analysis of Student Attention Levels

Artificial Intelligence-Based System for Detecting Attention Levels in Students

The attention level of students in a classroom can be improved through the use of Artificial Intelligence (AI) techniques. By automatically identifying the attention level, teachers can employ strategies to regain students' focus. This can be achieved through various sources of information.
One source is to analyze the emotions reflected on students' faces. AI can detect emotions, such as neutral, disgust, surprise, sadness, fear, happiness, and anger. Additionally, the direction of the students' gaze can also potentially indicate their level of attention. Another source is to observe the students' body posture. By using cameras and deep learning techniques, posture can be analyzed to determine the level of attention. For example, students who are slouching or resting their heads on their desks may have a lower level of attention. Smartwatches distributed to the students can provide biometric and other data, including heart rate and inertial measurements, which can also be used as indicators of attention. By combining these sources of information, an AI system can be trained to identify the level of attention in the classroom. However, integrating the different types of data poses a challenge that requires creating a labeled dataset. Expert input and existing studies are consulted for accurate labeling. In this paper, we propose the integration of such measurements and the creation of a dataset and a potential attention classifier. To provide feedback to the teacher, we explore various methods, such as smartwatches or direct computers. Once the teacher becomes aware of attention issues, they can adjust their teaching approach to re-engage and motivate the students. In summary, AI techniques can automatically identify the students' attention level by analyzing their emotions, gaze direction, body posture, and biometric data. This information can assist teachers in optimizing the teaching-learning process.

Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention

This paper proposes an artificial intelligence-based system to automatically detect whether students are paying attention to the class or are distracted. This system is designed to help teachers maintain students' attention, optimize their lessons, and dynamically introduce modifications in order for them to be more engaging.

The attention level of students in a classroom can be improved through the use of Artificial Intelligence (AI) techniques. By automatically identifying ...

Engineering

Capturing Dynamic Finger Gesturing with High-resolution Surface Electromyography and Computer Vision

Finger gestures are a critical element in human communication, and as such, finger gesture recognition is widely studied as a human-computer interface for state-of-the-art prosthetics and optimized rehabilitation. Surface electromyography (sEMG), in conjunction with deep learning methods, is considered a promising method in this domain. However, current methods often rely on cumbersome recording setups and the identification of static hand positions, limiting their effectiveness in real-world applications. The protocol we report here presents an advanced approach combining a wearable surface EMG and finger&#160;tracking system to capture comprehensive data during dynamic hand movements. The method records muscle activity from soft printed electrode arrays (16 electrodes) placed on the forearm as subjects perform gestures in different hand positions and during movement. Visual instructions prompt subjects to perform specific gestures while EMG and finger positions are recorded. The integration of synchronized EMG recordings and finger tracking data enables comprehensive analysis of muscle activity patterns and corresponding gestures. The reported approach demonstrates the potential of combining EMG and visual tracking technologies as an important resource for developing intuitive and responsive gesture recognition systems with applications in prosthetics, rehabilitation, and interactive technologies. This protocol aims to guide researchers and practitioners, fostering further innovation and application of gesture recognition in dynamic and real-world scenarios.

The paper presents a comprehensive protocol for simultaneously recording hand-electromyography (EMG) and visual finger tracking during natural finger gesturing. The visual data is designed to serve as the ground truth for the development of accurate EMG-based computational models for finger gesture recognition.

Finger gestures are a critical element in human communication, and as such, finger gesture recognition is widely studied as a human-computer interface for ...

Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation

Samenvatting

Meer video's verkennen