NOTE: A schematic detailing the protocol workflow can be seen in Figure 1.
1. 3D Scanning
<ol>
	<li>Obtain a 3D Scan of an oncologic resection specimen per the previously described 3D scanning protocol7.
	<ol>
		<li>Set up the commercially available, desktop 3D scanner in an area in proximity to the pathology lab where specimens are typically taken.</li>
		<li>Gently rinse and pat dry the resected specimen. Take 2D photographs of the anterior and posterior surfaces of the specimen.</li>
		<li>Place the specimen on the scanner turntable, ensuring the turntable is protected from any bodily fluids with a thin plastic sheet.</li>
		<li>Using the 3D scanner software and a laptop computer, obtain a 3D scan of the anterior and posterior surfaces of the specimen.</li>
		<li>Using the 3D scanner software, use three-point data cross-registration to align the two sets of scan data.</li>
		<li>Once the scan resembles the 3D structure of the ex vivo specimen, proceed to step 1.2.</li>
	</ol>
	</li>
	<li>Click Save Your Scan and save the completed scan from the 3D scanning software as a .obj file and a .jpeg file.</li>
</ol>
2. Augmented reality environment setup
<ol>
	<li>Open the desktop AR software. Open the Augmented Reality VISE package.</li>
	<li>Upload the 3D model produced in step 1.2 as a .obj file and the texture data as a .jpeg file into the software by dragging and dropping the files from the desktop into the Project Assets box.</li>
	<li>Put on the Head Mounted Display (HMD).</li>
	<li>Open the menu in the HMD environment by raising the non-dominant wrist. When a square icon appears, tap it by pointing the finger at the icon and look for a menu to appear.</li>
	<li>Select File | Show All Apps | open the Holographic Remoting application on the HMD.</li>
	<li>Connect the HMD to the software by selecting Mixed Reality | Remoting | Holographic Remoting for Play Mode in the top toolbar.</li>
	<li>Input the IP address of the HMD, which is displayed when the Holographic Remoting application is open. 
	NOTE: &#160;If not displayed within the app, the IP address can also be found in the settings or through the voice command &#8220;What&#8217;s My IP Address&#8221;.</li>
	<li>Ensure the HMD and the device running the software are both connected to the same Wi-Fi network. Hit the triangle Play button in the top center. The HMD headset will now display Rendering...</li>
	<li>Drag the 3D object model into the left Hierarchy bar. Open the dropdown menu for the object. Drag the default into the Adjust Pivot popup window under Replace Object. Click on Apply Selected Objects.</li>
	<li>Open the Medical_Object dropdown in the Hierarchy menu and select anchor before selecting Move Medical_Object&#8217;s pivot here in the Adjust Pivot popup.</li>
	<li>Drag the texture file into the (Texture 2D) box of the Adjust Pivot popup. Select Apply Selected Texture File.</li>
	<li>Hit the Play button in the top center of the screen once more.</li>
</ol>
3. Entering virtual reality
<ol>
	<li>Utilize the voice command Open to initialize the hologram.</li>
	<li>If the object is not seen or if it is ever lost from view, use the voice command Reposition to bring the object back to the center of view.</li>
	<li>The object will render in anatomical size. Use the voice command Lock Size to keep the object in the current size, Unlock Size to change the object size, and Back to Scale to restore the object to anatomical size.</li>
	<li>Pinch with the pointer finger and thumb to resize, move, and rotate the object in the same way a physical object is manipulated.</li>
</ol>
4. Utilizing voice commands
<ol>
	<li>Manipulate the object hands-free using the following voice commands:
	<ol>
		<li>Right: Rotate the object to the right.</li>
		<li>Left: Rotate the object to the left.</li>
		<li>Bigger: Make the object bigger.</li>
		<li>Smaller: Make the object smaller.</li>
		<li>Increase transparency: Increase the object&#39;s transparency.</li>
		<li>Decrease transparency: Decrease the object&#39;s transparency.</li>
	</ol>
	</li>
</ol>
5. AR-guided re-alignment of the specimen
NOTE: 3D realignment of the hologram can be seen in Figure 2.
<ol>
	<li>Manipulate the 3D specimen hologram seen through the HMD using the thumb and pointer finger.
	<ol>
		<li>Grasp the object by putting the thumb and pointer finger together in the real environment where the hologram is seen in the virtual environment.</li>
		<li>Let go of the object by opening the thumb and pointer finger in the real environment once the 3D hologram is placed where desired in the virtual environment.</li>
	</ol>
	</li>
	<li>Utilize the voice command Lock to secure the hologram in place and Unlock to manipulate the object once more.</li>
	<li>Based on the anatomic orientation of the hologram, realign the virtual specimen into the resection bed by following steps 5.1-5.2. When the hologram is realigned as much as possible with the resection bed as viewed through the HMD, use the voice command Lock to lock the specimen in place.</li>
</ol>

<ol>
	<li>Byers, R. M., Bland, K. I., Borlase, B., Luna, M. The prognostic and therapeutic value of frozen section determinations in the surgical treatment of squamous carcinoma of the head and neck. Am J Surg. 136 (4), 525-528 (1978).</li>
	<li>Ettl, T. et al. Positive frozen section margins predict local recurrence in R0-resected squamous cell carcinoma of the head and neck. Oral Oncol. 55, 17-23 (2016).</li>
	<li>Szewczyk, M. et al. Positive fresh frozen section margins as an adverse independent prognostic factor for local recurrence in oral cancer patients. Laryngoscope. 128 (5), 1093-1098 (2018).</li>
	<li>Kerawala, C. J., Ong, T. K. Relocating the site of frozen sections--is there room for improvement? Head Neck. 23 (3), 230-232 (2001).</li>
	<li>Coutu, B. et al. Positive margins matter regardless of subsequent resection findings. Oral Oncol. 128, 105850 (2022).</li>
	<li>Prasad, K. et al. How often is cancer present in oral cavity re-resections after initial positive margins? Laryngoscope. (2023).</li>
	<li>Prasad, K. et al. Augmented-reality surgery to guide head and neck cancer re-resection: a feasibility and accuracy study. Ann Surg Oncol. 30 (8), 4994-5000 (2023).</li>
	<li>Bopp, M. H. A. et al. Use of neuronavigation and augmented reality in transsphenoidal pituitary adenoma surgery. J Clin Med. 11 (19), 14 (2022).</li>
	<li>Checcucci, E. et al. The impact of 3D models on positive surgical margins after robot-assisted radical prostatectomy. World J Urol. 40 (9), 2221-2229 (2022).</li>
	<li>Gadzhiev, N. et al. Role and utility of mixed reality technology in laparoscopic partial nephrectomy: outcomes of a prospective RCT using an indigenously developed software. Adv Urol. 2022, 8992051 (2022).</li>
</ol>

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Digital Camera or Cameraphone</td>
			<td>iPhone&#160;</td>
			<td></td>
			<td>May use iPhone or similar camera phone or digital camera.</td>
		</tr>
		<tr>
			<td>EinScan SP V2 Platinum Desktop 3D Scanner</td>
			<td>Shining 3D</td>
			<td></td>
			<td>3D scanner hardware</td>
		</tr>
		<tr>
			<td>ExScan Software; Solid Edge SHINING 3D Edition</td>
			<td>Shining 3D</td>
			<td></td>
			<td>3D scanner software, included with purchase of 3D Scanner</td>
		</tr>
		<tr>
			<td>External Mouse</td>
			<td>Microsoft</td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Laptop Computer</td>
			<td>Dell XP5</td>
			<td>00355-60734-40310-AAOEM</td>
			<td>Laptop Requirements: 
			USB: 1 &#215;USB 2.0 or 3.0; OS: Win 7, 8 or 10 (64 bit); 
			Graphic Card: Nvidia series; Graphic memory: &#65310;1G; 
			CPU: Dual-core i5 or higher; Memory: &#65310;8G</td>
		</tr>
		<tr>
			<td>Microsoft HMD 2</td>
			<td>Microsoft</td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Unity Software</td>
			<td>Unity Technologies</td>
			<td></td>
			<td>Real-time, interactive 3D content platform</td>
		</tr>
		<tr>
			<td>Microsoft Office Suite</td>
			<td>Microsoft</td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Mobile Presentation Cart</td>
			<td>Oklahoma Sound</td>
			<td>PRC450</td>
			<td></td>
		</tr>
		<tr>
			<td>USB-c Device Converter</td>
			<td>TRIPP-LITE</td>
			<td>U442-DOCK3-B</td>
			<td>Necessary only if laptop does not have USB</td>
		</tr>
	</tbody>
</table>

augmented reality-based visualization of a resected tumor specimen

In this research protocol, we expand upon our previously described three-dimensional (3D) scanning protocol with the addition of augmented reality (AR) technology. Surgical margin status remains one of the most important prognostic features in cancers of the head and neck and is one of the few factors under the control of the head and neck surgeon1,2,3. The complex 3D anatomy of the head and neck often creates difficulty for surgeons attempting to relocate a positive or close margin, and prior studies have demonstrated that accuracy in relocating margins following initial resection is suboptimal4,5,6. As a result, oncologic outcomes following re-resection remain poor2,3. This protocol investigates the feasibility and accuracy of AR to guide the re-resection of initial positive or close margins in head and neck oncologic surgery. This technology allows surgeons to replace a 3D scanned virtual hologram of the resected cancer specimen back into the resection bed to improve visualization of re-resection location to adequately clear an initial positive or close margin.&#160;
ETHICS DISCLAIMER:
This protocol includes the use of cadavers with approval from the Vanderbilt University Medical Center&#8217;s Center for Experiential Learning and Assessment. This protocol was performed in compliance with Vanderbilt University Medical Center guidelines for human subjects research under IRB #221733.

This protocol was established in a cadaveric study7. Twenty cadaveric resections from the head and neck were completed; 13 (65%) cutaneous and 7 (35%) oral cavity resections were performed. The mean value of the greatest diameter of the resected specimens was 4.5 cm (range, 2.5&#8211;8.2 cm). The relocation error was (mean &#177; S.D.) 4 mm &#177; 3.9 mm (range, 1&#8211;15 mm). A significant difference was observed between the mean relocation error of maxillectomy and mandibulectomy specimens and all the other specimen types (10.7 vs 2.8 mm; p &#60; 0.01). The mean overall protocol time from the start of 3D scanning to alignment into the resection bed was 25.3 &#177; 8.9 min (range, 13.2&#8211;43.2 min). It took 11.5 &#177; 2.9 min (range, 6.9&#8211;20.0 min) to perform the 3D scanning and 2.7 &#177; 2.9 min (range, 0.3&#8211;22.5 min) to upload to the HMD after the scanning. From visualization within the HMD to the alignment of the model in the resection bed took 4.6 &#177; 2.4 min (range, 1.5&#8211;12.0 min). Relocation time based on specimen type did not differ significantly between skin and oral cavity (p = 0.51). These results are demonstrated in Table 1 and Table 27.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/66546/66546fig01.jpg" /> 
Figure 1: Protocol workflow. Schematic demonstrating the experimental protocols in parallel with the current standard protocol. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/66546/66546fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/66546/66546fig02.jpg" /> 
Figure 2: 3D hologram re-alignment viewed through the HMD. User view through the HMD of (A) the user&#8217;s hands grasping the 3D hologram, (B) the user&#8217;s hands releasing the 3D hologram onto the resection bed, and (C) the final product of the 3D hologram re-aligned into the resection. Abbreviation: HMD = head-mounted display. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/66546/66546fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td></td>
			<td>Anatomic subsite</td>
			<td>Specimen size (cm)</td>
			<td>Relocation error (mm)</td>
			<td>Time from start of 3D scan to alignment of hologram in resection bed (min)</td>
		</tr>
		<tr>
			<td>1</td>
			<td>Skin</td>
			<td>4.5&#215;4.0</td>
			<td>8</td>
			<td>28.5</td>
		</tr>
		<tr>
			<td>2</td>
			<td>Tongue</td>
			<td>7.0&#215;3.5</td>
			<td>1</td>
			<td>23.4</td>
		</tr>
		<tr>
			<td>3</td>
			<td>Skin</td>
			<td>5.0&#215;4.5</td>
			<td>7</td>
			<td>23.7</td>
		</tr>
		<tr>
			<td>4</td>
			<td>Skin</td>
			<td>7.0&#215;2.5</td>
			<td>1</td>
			<td>25.3</td>
		</tr>
		<tr>
			<td>5</td>
			<td>Maxilla</td>
			<td>4.0&#215;2.5</td>
			<td>6</td>
			<td>15.4</td>
		</tr>
		<tr>
			<td>6</td>
			<td>Skin</td>
			<td>7.0&#215;4.0</td>
			<td>3</td>
			<td>14.8</td>
		</tr>
		<tr>
			<td>7</td>
			<td>Lip</td>
			<td>5.5&#215;2.5</td>
			<td>2</td>
			<td>33.9</td>
		</tr>
		<tr>
			<td>8</td>
			<td>Skin</td>
			<td>4.0&#215;4.0</td>
			<td>4</td>
			<td>17.1</td>
		</tr>
		<tr>
			<td>9</td>
			<td>Skin</td>
			<td>6.0&#215;4.0</td>
			<td>7</td>
			<td>17.6</td>
		</tr>
		<tr>
			<td>10</td>
			<td>Skin</td>
			<td>4.0&#215;2.5</td>
			<td>2</td>
			<td>20.6</td>
		</tr>
		<tr>
			<td>11</td>
			<td>Skin</td>
			<td>6.0&#215;5.0</td>
			<td>1</td>
			<td>13.2</td>
		</tr>
		<tr>
			<td>12</td>
			<td>Skin</td>
			<td>4.4&#215;2.8</td>
			<td>2</td>
			<td>37.7</td>
		</tr>
		<tr>
			<td>13</td>
			<td>Skin</td>
			<td>3.5&#215;3.0</td>
			<td>1</td>
			<td>17.3</td>
		</tr>
		<tr>
			<td>14</td>
			<td>Skin</td>
			<td>4.3&#215;3.9</td>
			<td>4</td>
			<td>43.2</td>
		</tr>
		<tr>
			<td>15</td>
			<td>Skin</td>
			<td>3.0&#215;3.0</td>
			<td>2</td>
			<td>35.3</td>
		</tr>
		<tr>
			<td>16</td>
			<td>Lip</td>
			<td>3.0&#215;2.8</td>
			<td>1</td>
			<td>31.7</td>
		</tr>
		<tr>
			<td>17</td>
			<td>Tongue</td>
			<td>5.2&#215;3.0</td>
			<td>1</td>
			<td>33.8</td>
		</tr>
		<tr>
			<td>18</td>
			<td>Maxilla</td>
			<td>7.0&#215;3.5</td>
			<td>11</td>
			<td>23.4</td>
		</tr>
		<tr>
			<td>19</td>
			<td>Skin</td>
			<td>8.2&#215;5.2</td>
			<td>1</td>
			<td>28.5</td>
		</tr>
		<tr>
			<td>20</td>
			<td>Mandible</td>
			<td>7.3&#215;6.1</td>
			<td>15</td>
			<td>23.4</td>
		</tr>
		<tr>
			<td></td>
			<td>Mean&#177; SD</td>
			<td></td>
			<td>&#160;4.0 &#177; 3.9</td>
			<td>25.3 &#177; 8.9</td>
		</tr>
	</tbody>
</table>
Table 1: Resection specimen characteristics and relocation error. Reproduced with permission from Prasad et al7.
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td colspan="2">Interval</td>
			<td colspan="2">Start point</td>
			<td colspan="2">End point</td>
			<td>Mean 
			(min)</td>
			<td>Standard deviation 
			(min)</td>
			<td>Range 
			(min)</td>
		</tr>
		<tr>
			<td colspan="2">Specimen 3D scanning image acquisition</td>
			<td colspan="2">3D scan initiated</td>
			<td colspan="2">3D scan exported</td>
			<td>11.5</td>
			<td>2.9</td>
			<td>6.9-20.0</td>
		</tr>
		<tr>
			<td colspan="2">Uploading to Hololens</td>
			<td colspan="2">3D scan exported</td>
			<td colspan="2">3D model uploaded</td>
			<td>2.7</td>
			<td>2.9</td>
			<td>0.3-12.5</td>
		</tr>
		<tr>
			<td colspan="2">Visualization in Hololens</td>
			<td colspan="2">3D model uploaded</td>
			<td colspan="2">Specimen hologram visible in AR</td>
			<td>6.3</td>
			<td>5.7</td>
			<td>0.3-22.5</td>
		</tr>
		<tr>
			<td colspan="2">Hologram alignment</td>
			<td colspan="2">Specimen hologram visible in&#160;AR</td>
			<td colspan="2">Specimen hologram aligned to resection&#160;bed</td>
			<td>4.6</td>
			<td>2.4</td>
			<td>1.5-12.0</td>
		</tr>
		<tr>
			<td colspan="2">Full workflow</td>
			<td colspan="2">3D scan initiated</td>
			<td colspan="2">Specimen hologram aligned to resection bed</td>
			<td>25.3</td>
			<td>8.9</td>
			<td>13.2-43.2</td>
		</tr>
		<tr>
			<td colspan="9">3D Three dimensional; AR Augmented reality</td>
		</tr>
	</tbody>
</table>
Table 2: Time interval analysis. Reproduced with permission from Prasad et al7.

Augmented Reality-Based Visualization of a Resected Tumor Specimen

NOTE: A schematic detailing the protocol workflow can be seen in Figure 1.
1. 3D Scanning

	Obtain a 3D Scan of an oncologic resection specimen per the ...

To begin, gently pat dry a rinsed, resected oncologic specimen. Take 2D photographs of the anterior and posterior surfaces of the specimen. 
Place the specimen on a 3D scanner&#39;s plastic-covered turntable. Scan the specimen&#39;s anterior and posterior surfaces to obtain a 3D image. 
Use three-point data cross-registration to align the two sets of scan data. When the scan results resemble the 3D ex vivo specimen, click on Save Your Scan to save the scan as a .obj and a .jpeg file. 
Next launch the Augmented Reality software and open the Augmented Reality VISE package. Drag and drop the .obj and .jpeg files from the desktop into Project Assets box.&#160; &#160;
Put on the Head Mounted Display or HMD and raise the non-dominant wrist to open the menu in the HMD environment. When a square icon appears, point the finger at the icon and tap.
When the menu appears, click on File followed by Show All Apps and open the Holographic Remoting application. In the top toolbar, select Mixed Reality then press Remoting and Holographic Remoting for Play Mode. 
Input the IP address of the HMD, then hit Play after ensuring that the HMD and device are connected to the same WiFi network. The headset should now display Receiving&#8230;. 
Drag the 3D object model into the left Hierarchy bar, open the dropdown menu, and drag the default into the Adjust Pivot popup window under Replace Object. 
Now click on Apply Selected Objects then open the Medical_Object dropdown in the Hierarchy menu and choose anchor. Choose Move Medical_Object&#39;s pivot here in the Adjust Pivot popup. 
Drag the texture file into the (Texture 2D) box of the Adjust Pivot popup, select Apply Selected Texture File, and hit the Play button. 
Now use the voice commands Open to initialize the hologram, Reposition to bring the object into the center of the view if it is not visible, Lock Size to keep the object in the current size, Unlock Size to change the object size, and Back to Scale to restore the object to anatomical size. 
Pinch with the pointer finger and thumb to resize, move, and rotate the object. Control the object hands-free using voice commands Right or Left to rotate, Bigger or Smaller to resize, and Increase or Decrease transparency to adjust its transparency. 
To realign the hologram, place the thumb and pointer finger together to grasp the object, then realign it against the resection bed, and release the object in the desired virtual spot. Finally, use the voice commands Lock or Unlock to secure the hologram in its place.

augmented-reality-based-visualization-of-a-resected-tumor-specimen

Nanyang Technological University

Research

JoVE Encyclopedia of Experiments

Cancer Research

Head and Neck Cancer

Assays & techniques

EoE Sub Articles

eoe sub articles

All procedures involving sample collection have been performed in accordance with the institute&#39;s IRB guidelines.
NOTE: A schematic detailing the protocol workflow can be seen in Figure 1.
1. 3D Scanning
<ol>
	<li>Obtain a 3D Scan of an oncologic resection specimen per the previously described 3D scanning protocol7.
	<ol>
		<li>Set up the commercially available desktop 3D scanner in proximity to the pathology lab where specimens are typically taken.</li>
		<li>Gently rinse, and pat dry the resected specimen. Then, take 2D photographs of its anterior and posterior surfaces.</li>
		<li>Place the specimen on the scanner turntable, ensuring the turntable is protected from any bodily fluids with a thin plastic sheet.</li>
		<li>Using 3D scanner software and a laptop computer, obtain a 3D scan of the specimen&#39;s anterior and posterior surfaces.</li>
		<li>Using the 3D scanner software, use three-point data cross-registration to align the two sets of scan data.</li>
		<li>Once the scan resembles the 3D structure of the ex vivo specimen, proceed to step 1.2.</li>
	</ol>
	</li>
	<li>Click Save Your Scan and save the completed scan from the 3D scanning software as a .obj file and a .jpeg file.</li>
</ol>
2. Augmented reality environment setup
<ol>
	<li>Open the desktop AR software. Open the Augmented Reality VISE package.</li>
	<li>Upload the 3D model produced in step 1.2 as a .obj file and the texture data as a .jpeg file into the software by dragging and dropping the files from the desktop into the Project Assets box.</li>
	<li>Put on the Head Mounted Display (HMD).</li>
	<li>Open the menu in the HMD environment by raising the non-dominant wrist. When a square icon appears, tap it by pointing the finger at the icon and look for a menu to appear.</li>
	<li>Select File | Show All Apps | open the Holographic Remoting application on the HMD.</li>
	<li>Connect the HMD to the software by selecting Mixed Reality | Remoting | Holographic Remoting for Play Mode in the top toolbar.</li>
	<li>Input the IP address of the HMD, which is displayed when the Holographic Remoting application is open. 
	NOTE:&#160; If not displayed within the app, the IP address can also be found in the settings or through the voice command &#34;What&#39;s My IP Address&#34;.</li>
	<li>Ensure the HMD and the device running the software are both connected to the same Wi-Fi network. Hit the triangle Play button in the top center. The HMD headset will now display Rendering.</li>
	<li>Drag the 3D object model into the left Hierarchy bar. Open the dropdown menu for the object. Drag the default into the Adjust Pivot popup window under Replace Object. Click on Apply Selected Objects.</li>
	<li>Open the Medical_Object dropdown in the Hierarchy menu and select anchor before selecting Move Medical_Object&#39;s pivot here in the Adjust Pivot popup.</li>
	<li>Drag the texture file into the (Texture 2D) box of the Adjust Pivot popup. Select Apply Selected Texture File.</li>
	<li>Hit the Play button in the top center of the screen once more.</li>
</ol>
3. Entering virtual reality
<ol>
	<li>Utilize the voice command Open to initialize the hologram.</li>
	<li>If the object is not seen or if it is ever lost from view, use the voice command Reposition to bring the object back to the center of view.</li>
	<li>The object will render in anatomical size. Use the voice command Lock Size to keep the object in the current size, Unlock Size to change the object size, and Back to Scale to restore the object to anatomical size.</li>
	<li>Pinch with the pointer finger and thumb to resize, move, and rotate the object in the same way a physical object is manipulated.</li>
</ol>
4. Utilizing voice commands
<ol>
	<li>Manipulate the object hands-free using the following voice commands:
	<ol>
		<li>Right: Rotate the object to the right.</li>
		<li>Left: Rotate the object to the left.</li>
		<li>Bigger: Make the object bigger.</li>
		<li>Smaller: Make the object smaller.</li>
		<li>Increase transparency: Increase the object&#39;s transparency.</li>
		<li>Decrease transparency: Decrease the object&#39;s transparency.</li>
	</ol>
	</li>
</ol>
5. AR-guided re-alignment of the specimen
NOTE: 3D realignment of the hologram can be seen in Figure 2.
<ol>
	<li>Manipulate the 3D specimen hologram seen through the HMD using the thumb and pointer finger.
	<ol>
		<li>Grasp the object by putting the thumb and pointer finger together in the real environment where the hologram is seen in the virtual environment.</li>
		<li>Let go of the object by opening the thumb and pointer finger in the real environment once the 3D hologram is placed where desired in the virtual environment.</li>
	</ol>
	</li>
	<li>Utilize the voice command Lock to secure the hologram in place and Unlock to manipulate the object once more.</li>
	<li>Based on the anatomic orientation of the hologram, realign the virtual specimen into the resection bed by following steps 5.1-5.2. When the hologram is realigned as much as possible with the resection bed as viewed through the HMD, use the voice command Lock to lock the specimen in place.</li>
</ol>

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Digital Camera or Cameraphone</td>
			<td>iPhone</td>
			<td></td>
			<td>May use iPhone or similar camera phone or digital camera.</td>
		</tr>
		<tr>
			<td>EinScan SP V2 Platinum Desktop 3D Scanner</td>
			<td>Shining 3D</td>
			<td></td>
			<td>3D scanner hardware</td>
		</tr>
		<tr>
			<td>ExScan Software; Solid Edge SHINING 3D Edition</td>
			<td>Shining 3D</td>
			<td></td>
			<td>3D scanner software, included with purchase of 3D Scanner</td>
		</tr>
		<tr>
			<td>External Mouse</td>
			<td>Microsoft</td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Laptop Computer</td>
			<td>Dell XP5</td>
			<td>00355-60734-40310-AAOEM</td>
			<td>Laptop Requirements:</td>
		</tr>
		<tr>
			<td>USB: 1 &#215;USB 2.0 or 3.0; OS: Win 7, 8 or 10 (64 bit);</td>
			<td></td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Graphic Card: Nvidia series; Graphic memory: ?1G;</td>
			<td></td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>CPU: Dual-core i5 or higher; Memory: ?8G</td>
			<td></td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Microsoft HMD 2</td>
			<td>Microsoft</td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Unity Software</td>
			<td>Unity Technologies</td>
			<td></td>
			<td>Real-time, interactive 3D content platform</td>
		</tr>
		<tr>
			<td>Microsoft Office Suite</td>
			<td>Microsoft</td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Mobile Presentation Cart</td>
			<td>Oklahoma Sound</td>
			<td>PRC450</td>
			<td></td>
		</tr>
		<tr>
			<td>USB-c Device Converter</td>
			<td>TRIPP-LITE</td>
			<td>U442-DOCK3-B</td>
			<td>Necessary only if laptop does not have USB</td>
		</tr>
	</tbody>
</table>

This video demonstrates using augmented reality (AR) for tumor resection, where a 3D-scanned tumor specimen is visualized as a hologram on a head-mounted display. Surgeons can manipulate and lock the hologram in place to ensure precise tumor removal.

<img alt="Figure 1" src="/files/ftp_upload/23375/23375_Figure1.jpg" /> 
Figure 1: Protocol workflow. Schematic demonstrating the experimental protocols in parallel with the current standard protocol.
<img alt="Figure 2" src="/files/ftp_upload/23375/23375_Figure2.jpg" /> 
Figure 2: 3D hologram re-alignment viewed through the HMD. User view through the HMD of (A) the user&#39;s hands grasping the 3D hologram, (B) the user&#39;s hands releasing the 3D hologram onto the resection bed, and (C) the final product of the 3D hologram re-aligned into the resection. Abbreviation: HMD = head-mounted display.

All procedures involving sample collection have been performed in accordance with the institute&#39;s IRB guidelines.
NOTE: A schematic detailing the ...

Place a resected tumor specimen on a 3D scanning table connected to a computer. Scan the anterior and posterior surfaces to create a 3D image.
Once this 3D image matches the physical specimen, open it in augmented reality or AR software.
Wear a head-mounted display or HMD and connect it wirelessly to the AR software.
This allows visualization of the 3D image through the HMD device as a virtual hologram of the specimen.
Align and anchor the hologram to ensure spatial accuracy within the AR environment, then apply textures to enhance realism.
Use a voice command to open the hologram in a virtual environment and, with hand gestures, reposition, rotate, and adjust the size.
Alternatively, use voice commands for hands-free adjustments.
This augmented reality-based visualization generates a tumor hologram that aids in precise margin identification at the surgery site.

88 Views. בפרוטוקול מחקר זה, אנו מרחיבים את פרוטוקול הסריקה התלת-ממדי (3D) שתואר קודם לכן עם תוספת של טכנולוגיית מציאות רבודה (AR). מצב שוליים ניתוחיים נותר אחד המאפיינים הפרוגנוסטיים החשובים ביותר בסרטן הראש והצוואר והוא אחד הגורמים הבודדים הנמצאים תחת שליטתו של מנתח הראש והצוואר1,2,3. הא...

Video: הדמיה מבוססת מציאות רבודה של דגימת גידול שנותחה - Experiment

Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study

This study introduces an experimental paradigm for a usability test of emerging technologies in a management information system (MIS). The usability test included both subjective and objective evaluations. For the subjective evaluation, a usability questionnaire and a NASA-TLX scale were adopted. For the objective evaluation, methods of Neuro-Information-Systems (NeuroIS) were used. From a NeuroIS perspective, this study used mobile fNIRS and eye tracking glasses for multimodal measurements, which solved the problem of ecological validity of cognitive neuroscience tools used in real-world behavior experiments. This study used Augmented Reality (AR) integrated into the Internet of Things (IoT) as an experimental object. Comparing the differences in the neuroimaging data, the physiological data, the usability questionnaire, and the NASA-TLX scale data between the two information-search modes (AR versus a website), information search with AR had a higher efficiency and a lower cognitive load compared with information search with a website during the process of consumption decision-making. The usability experiment results demonstrate that AR, as an emerging technology in retail, can effectively enhance consumer experiences and increase their purchase intention. The experimental paradigm, combining both subjective and objective evaluations in this study, could be applied to a usability test for emerging technologies, such as augmented reality, virtual reality, artificial intelligence, wearable technology, robotics, and big data. It provides a practical experimental solution for the user experience in human-computer-interactions with the adoption of emerging technologies.

This study presents an experimental paradigm for&#160;a&#160;usability test combining subjective and objective evaluations. The objective evaluation adopted Neuro-Information-Systems (NeuroIS) methods, and the subjective evaluation adopted a usability questionnaire and a NASA-Task Load Index (NASA-TLX) scale.

הערכת שימושיות של מציאות רבודה: מחקר נוירו-אינפורמציה-מערכות

This study introduces an experimental paradigm for a usability test of emerging technologies in a management information system (MIS). The usability test ...

JoVE Journal

Behavior

Augmented Reality Navigation-Guided Core Decompression for Osteonecrosis of Femoral Head

Osteonecrosis of the femoral head (ONFH) is a common joint disease in young and middle-aged patients, which seriously burdens their lives and work. For early-stage ONFH, core decompression surgery is a classical and effective hip preservation therapy. In traditional procedures of core decompression with Kirschner wire, there are still many problems such as X-ray exposure, repeated puncture verification, and damage to normal bone tissue. The blindness of the puncture process and the inability to provide real-time visualization are crucial reasons for these problems.
To optimize this procedure, our team developed an intraoperative navigation system on the basis of augmented reality (AR) technology. This surgical system can intuitively display the anatomy of the surgical areas and render preoperative images and virtual needles to intraoperative video in real-time. With the guide of the navigation system, surgeons can accurately insert Kirschner wires into the targeted lesion area and minimize the collateral damage. We conducted 10 cases of core decompression surgery with this system. The efficiency of positioning and fluoroscopy is greatly improved compared to the traditional procedures, and the accuracy of puncture is also guaranteed.

Augmented reality technology was applied to core decompression for osteonecrosis of the femoral head to realize real-time visualization of this surgical procedure. This method can effectively improve the safety and precision of core decompression.

מציאות רבודה ניווט מונחה ניווט דקומפרסיה ליבה עבור אוסטאונקרוזיס של ראש הירך

Osteonecrosis of the femoral head (ONFH) is a common joint disease in young and middle-aged patients, which seriously burdens their lives and work. For ...

Medicine

שחזור תלת-ממדי אופטימלי ומיפוי תנועה לניתוחים מרחוק בסיוע AR

Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter Aortic Valve Replacement (TAVR). TAVR is the alternate medical procedure for open-heart surgery. TAVR replaces the injured valve with the new one using a catheter. In the existing model, remote guidance is given, while the surgery is not automated based on AR. In this article, we deployed a spatially aligned camera that is connected to a motor for the automation of image capture in the surgical environment. The camera tracks the 2D high-resolution image of the patient's heart along with the catheter testbed. These captured images are uploaded using the mobile app to a remote surgeon who is a cardiology expert. This image is utilized for the 3D reconstruction from 2D image tracking. This is viewed in a HoloLens like an emulator in a laptop. The surgeon can remotely inspect the 3D reconstructed images with additional transformation features such as rotation and scaling. These transformation features are enabled through hand gestures. The surgeon's guidance is transmitted to the surgical environment to automate the process in real-time scenarios. The catheter testbed in the surgical field is controlled by the hand gesture guidance of the remote surgeon. The developed prototype model demonstrates the effectiveness of remote surgical guidance through AR.

מחבר Spotlight: מהפכה בניתוחים מרחוק עם מציאות רבודה ורובוטיקה לשיפור הדיוק והנגישות

This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity.

ניתוח אוטומטי בהחלפת מסתם אבי העורקים באמצעות צנתר במציאות רבודה

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter ...

Augmented Reality-Based Visualization of a Resected Tumor Specimen

Transcript

Augmented Reality-Based Visualization of a Resected Tumor Specimen