Name: Video: Augmented Reality-Based Visualization of a Resected Tumor Specimen - Concept
Uploaded: 2025-03-04T15:35:02.000+00:00
Duration: 1 min 11 s
Description: 70 Views. This video demonstrates using augmented reality (AR) for tumor resection, where a 3D-scanned tumor specimen is visualized as a hologram on a head-mounted display. Surgeons can manipulate and lock the hologram in place to ensure precise tumor removal.

eoe sub articles

EoE Sub Articles

All procedures involving sample collection have been performed in accordance with the institute&#39;s IRB guidelines.
NOTE: A schematic detailing the protocol workflow can be seen in Figure 1.
1. 3D Scanning
<ol>
	<li>Obtain a 3D Scan of an oncologic resection specimen per the previously described 3D scanning protocol7.
	<ol>
		<li>Set up the commercially available desktop 3D scanner in proximity to the pathology lab where specimens are typically taken.</li>
		<li>Gently rinse, and pat dry the resected specimen. Then, take 2D photographs of its anterior and posterior surfaces.</li>
		<li>Place the specimen on the scanner turntable, ensuring the turntable is protected from any bodily fluids with a thin plastic sheet.</li>
		<li>Using 3D scanner software and a laptop computer, obtain a 3D scan of the specimen&#39;s anterior and posterior surfaces.</li>
		<li>Using the 3D scanner software, use three-point data cross-registration to align the two sets of scan data.</li>
		<li>Once the scan resembles the 3D structure of the ex vivo specimen, proceed to step 1.2.</li>
	</ol>
	</li>
	<li>Click Save Your Scan and save the completed scan from the 3D scanning software as a .obj file and a .jpeg file.</li>
</ol>
2. Augmented reality environment setup
<ol>
	<li>Open the desktop AR software. Open the Augmented Reality VISE package.</li>
	<li>Upload the 3D model produced in step 1.2 as a .obj file and the texture data as a .jpeg file into the software by dragging and dropping the files from the desktop into the Project Assets box.</li>
	<li>Put on the Head Mounted Display (HMD).</li>
	<li>Open the menu in the HMD environment by raising the non-dominant wrist. When a square icon appears, tap it by pointing the finger at the icon and look for a menu to appear.</li>
	<li>Select File | Show All Apps | open the Holographic Remoting application on the HMD.</li>
	<li>Connect the HMD to the software by selecting Mixed Reality | Remoting | Holographic Remoting for Play Mode in the top toolbar.</li>
	<li>Input the IP address of the HMD, which is displayed when the Holographic Remoting application is open. 
	NOTE:&#160; If not displayed within the app, the IP address can also be found in the settings or through the voice command &#34;What&#39;s My IP Address&#34;.</li>
	<li>Ensure the HMD and the device running the software are both connected to the same Wi-Fi network. Hit the triangle Play button in the top center. The HMD headset will now display Rendering.</li>
	<li>Drag the 3D object model into the left Hierarchy bar. Open the dropdown menu for the object. Drag the default into the Adjust Pivot popup window under Replace Object. Click on Apply Selected Objects.</li>
	<li>Open the Medical_Object dropdown in the Hierarchy menu and select anchor before selecting Move Medical_Object&#39;s pivot here in the Adjust Pivot popup.</li>
	<li>Drag the texture file into the (Texture 2D) box of the Adjust Pivot popup. Select Apply Selected Texture File.</li>
	<li>Hit the Play button in the top center of the screen once more.</li>
</ol>
3. Entering virtual reality
<ol>
	<li>Utilize the voice command Open to initialize the hologram.</li>
	<li>If the object is not seen or if it is ever lost from view, use the voice command Reposition to bring the object back to the center of view.</li>
	<li>The object will render in anatomical size. Use the voice command Lock Size to keep the object in the current size, Unlock Size to change the object size, and Back to Scale to restore the object to anatomical size.</li>
	<li>Pinch with the pointer finger and thumb to resize, move, and rotate the object in the same way a physical object is manipulated.</li>
</ol>
4. Utilizing voice commands
<ol>
	<li>Manipulate the object hands-free using the following voice commands:
	<ol>
		<li>Right: Rotate the object to the right.</li>
		<li>Left: Rotate the object to the left.</li>
		<li>Bigger: Make the object bigger.</li>
		<li>Smaller: Make the object smaller.</li>
		<li>Increase transparency: Increase the object&#39;s transparency.</li>
		<li>Decrease transparency: Decrease the object&#39;s transparency.</li>
	</ol>
	</li>
</ol>
5. AR-guided re-alignment of the specimen
NOTE: 3D realignment of the hologram can be seen in Figure 2.
<ol>
	<li>Manipulate the 3D specimen hologram seen through the HMD using the thumb and pointer finger.
	<ol>
		<li>Grasp the object by putting the thumb and pointer finger together in the real environment where the hologram is seen in the virtual environment.</li>
		<li>Let go of the object by opening the thumb and pointer finger in the real environment once the 3D hologram is placed where desired in the virtual environment.</li>
	</ol>
	</li>
	<li>Utilize the voice command Lock to secure the hologram in place and Unlock to manipulate the object once more.</li>
	<li>Based on the anatomic orientation of the hologram, realign the virtual specimen into the resection bed by following steps 5.1-5.2. When the hologram is realigned as much as possible with the resection bed as viewed through the HMD, use the voice command Lock to lock the specimen in place.</li>
</ol>

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Digital Camera or Cameraphone</td>
			<td>iPhone</td>
			<td></td>
			<td>May use iPhone or similar camera phone or digital camera.</td>
		</tr>
		<tr>
			<td>EinScan SP V2 Platinum Desktop 3D Scanner</td>
			<td>Shining 3D</td>
			<td></td>
			<td>3D scanner hardware</td>
		</tr>
		<tr>
			<td>ExScan Software; Solid Edge SHINING 3D Edition</td>
			<td>Shining 3D</td>
			<td></td>
			<td>3D scanner software, included with purchase of 3D Scanner</td>
		</tr>
		<tr>
			<td>External Mouse</td>
			<td>Microsoft</td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Laptop Computer</td>
			<td>Dell XP5</td>
			<td>00355-60734-40310-AAOEM</td>
			<td>Laptop Requirements:</td>
		</tr>
		<tr>
			<td>USB: 1 &#215;USB 2.0 or 3.0; OS: Win 7, 8 or 10 (64 bit);</td>
			<td></td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Graphic Card: Nvidia series; Graphic memory: ?1G;</td>
			<td></td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>CPU: Dual-core i5 or higher; Memory: ?8G</td>
			<td></td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Microsoft HMD 2</td>
			<td>Microsoft</td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Unity Software</td>
			<td>Unity Technologies</td>
			<td></td>
			<td>Real-time, interactive 3D content platform</td>
		</tr>
		<tr>
			<td>Microsoft Office Suite</td>
			<td>Microsoft</td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Mobile Presentation Cart</td>
			<td>Oklahoma Sound</td>
			<td>PRC450</td>
			<td></td>
		</tr>
		<tr>
			<td>USB-c Device Converter</td>
			<td>TRIPP-LITE</td>
			<td>U442-DOCK3-B</td>
			<td>Necessary only if laptop does not have USB</td>
		</tr>
	</tbody>
</table>

augmented reality-based visualization of a resected tumor specimen

This video demonstrates using augmented reality (AR) for tumor resection, where a 3D-scanned tumor specimen is visualized as a hologram on a head-mounted display. Surgeons can manipulate and lock the hologram in place to ensure precise tumor removal.

<img alt="Figure 1" src="/files/ftp_upload/23375/23375_Figure1.jpg" /> 
Figure 1: Protocol workflow. Schematic demonstrating the experimental protocols in parallel with the current standard protocol.
<img alt="Figure 2" src="/files/ftp_upload/23375/23375_Figure2.jpg" /> 
Figure 2: 3D hologram re-alignment viewed through the HMD. User view through the HMD of (A) the user&#39;s hands grasping the 3D hologram, (B) the user&#39;s hands releasing the 3D hologram onto the resection bed, and (C) the final product of the 3D hologram re-aligned into the resection. Abbreviation: HMD = head-mounted display.

Augmented Reality-Based Visualization of a Resected Tumor Specimen

All procedures involving sample collection have been performed in accordance with the institute&#39;s IRB guidelines.
NOTE: A schematic detailing the ...

Place a resected tumor specimen on a 3D scanning table connected to a computer. Scan the anterior and posterior surfaces to create a 3D image.
Once this 3D image matches the physical specimen, open it in augmented reality or AR software.
Wear a head-mounted display or HMD and connect it wirelessly to the AR software.
This allows visualization of the 3D image through the HMD device as a virtual hologram of the specimen.
Align and anchor the hologram to ensure spatial accuracy within the AR environment, then apply textures to enhance realism.
Use a voice command to open the hologram in a virtual environment and, with hand gestures, reposition, rotate, and adjust the size.
Alternatively, use voice commands for hands-free adjustments.
This augmented reality-based visualization generates a tumor hologram that aids in precise margin identification at the surgery site.

augmented-reality-based-visualization-of-a-resected-tumor-specimen

Nanyang Technological University

Research

JoVE Encyclopedia of Experiments

Cancer Research

Head and Neck Cancer

Assays & techniques

70 Views. This video demonstrates using augmented reality (AR) for tumor resection, where a 3D-scanned tumor specimen is visualized as a hologram on a head-mounted display. Surgeons can manipulate and lock the hologram in place to ensure precise tumor removal. 

Video: Augmented Reality-Based Visualization of a Resected Tumor Specimen - Concept

Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study

This study introduces an experimental paradigm for a usability test of emerging technologies in a management information system (MIS). The usability test included both subjective and objective evaluations. For the subjective evaluation, a usability questionnaire and a NASA-TLX scale were adopted. For the objective evaluation, methods of Neuro-Information-Systems (NeuroIS) were used. From a NeuroIS perspective, this study used mobile fNIRS and eye tracking glasses for multimodal measurements, which solved the problem of ecological validity of cognitive neuroscience tools used in real-world behavior experiments. This study used Augmented Reality (AR) integrated into the Internet of Things (IoT) as an experimental object. Comparing the differences in the neuroimaging data, the physiological data, the usability questionnaire, and the NASA-TLX scale data between the two information-search modes (AR versus a website), information search with AR had a higher efficiency and a lower cognitive load compared with information search with a website during the process of consumption decision-making. The usability experiment results demonstrate that AR, as an emerging technology in retail, can effectively enhance consumer experiences and increase their purchase intention. The experimental paradigm, combining both subjective and objective evaluations in this study, could be applied to a usability test for emerging technologies, such as augmented reality, virtual reality, artificial intelligence, wearable technology, robotics, and big data. It provides a practical experimental solution for the user experience in human-computer-interactions with the adoption of emerging technologies.

This study presents an experimental paradigm for&#160;a&#160;usability test combining subjective and objective evaluations. The objective evaluation adopted Neuro-Information-Systems (NeuroIS) methods, and the subjective evaluation adopted a usability questionnaire and a NASA-Task Load Index (NASA-TLX) scale.

This study introduces an experimental paradigm for a usability test of emerging technologies in a management information system (MIS). The usability test ...

JoVE Journal

Behavior

Augmented Reality Navigation-Guided Core Decompression for Osteonecrosis of Femoral Head

Osteonecrosis of the femoral head (ONFH) is a common joint disease in young and middle-aged patients, which seriously burdens their lives and work. For early-stage ONFH, core decompression surgery is a classical and effective hip preservation therapy. In traditional procedures of core decompression with Kirschner wire, there are still many problems such as X-ray exposure, repeated puncture verification, and damage to normal bone tissue. The blindness of the puncture process and the inability to provide real-time visualization are crucial reasons for these problems.
To optimize this procedure, our team developed an intraoperative navigation system on the basis of augmented reality (AR) technology. This surgical system can intuitively display the anatomy of the surgical areas and render preoperative images and virtual needles to intraoperative video in real-time. With the guide of the navigation system, surgeons can accurately insert Kirschner wires into the targeted lesion area and minimize the collateral damage. We conducted 10 cases of core decompression surgery with this system. The efficiency of positioning and fluoroscopy is greatly improved compared to the traditional procedures, and the accuracy of puncture is also guaranteed.

Augmented reality technology was applied to core decompression for osteonecrosis of the femoral head to realize real-time visualization of this surgical procedure. This method can effectively improve the safety and precision of core decompression.

Osteonecrosis of the femoral head (ONFH) is a common joint disease in young and middle-aged patients, which seriously burdens their lives and work. For ...

Medicine

Optimized 3D Reconstruction and Motion Mapping for Remote AR-Assisted Surgery

Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter Aortic Valve Replacement (TAVR). TAVR is the alternate medical procedure for open-heart surgery. TAVR replaces the injured valve with the new one using a catheter. In the existing model, remote guidance is given, while the surgery is not automated based on AR. In this article, we deployed a spatially aligned camera that is connected to a motor for the automation of image capture in the surgical environment. The camera tracks the 2D high-resolution image of the patient's heart along with the catheter testbed. These captured images are uploaded using the mobile app to a remote surgeon who is a cardiology expert. This image is utilized for the 3D reconstruction from 2D image tracking. This is viewed in a HoloLens like an emulator in a laptop. The surgeon can remotely inspect the 3D reconstructed images with additional transformation features such as rotation and scaling. These transformation features are enabled through hand gestures. The surgeon's guidance is transmitted to the surgical environment to automate the process in real-time scenarios. The catheter testbed in the surgical field is controlled by the hand gesture guidance of the remote surgeon. The developed prototype model demonstrates the effectiveness of remote surgical guidance through AR.

Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility

This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity.

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter ...

Augmented Reality-Based Visualization of a Resected Tumor Specimen

Transcript

Augmented Reality-Based Visualization of a Resected Tumor Specimen

Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study

Augmented Reality Navigation-Guided Core Decompression for Osteonecrosis of Femoral Head

Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality