This study was supported by the Penn State College of Medicine and the Department of Biochemistry and Molecular Biology, as well as Tobacco Settlement Fund (TSF) grant 4100079742-EXT. The CryoEM and CryoET Core (RRID:SCR_021178) services and instruments used in this project were funded, in part, by the Pennsylvania State University College of Medicine via the Office of the Vice Dean of Research and Graduate Students and the Pennsylvania Department of Health using Tobacco Settlement Funds (CURE). The content is solely the responsibility of the authors and does not necessarily represent the official views of the University or College of Medicine. The Pennsylvania Department of Health specifically disclaims responsibility for any analyses, interpretations, or conclusions.

<ol>
	<li>Bai, X. -. C., Mcmullan, G., Scheres, S. H. W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=How+cryo-EM+is+revolutionizing+structural+biology.">How cryo-EM is revolutionizing structural biology.</a> Trends in Biochemical Sciences. 40 (1), 49-57 (2015).</li><li>de Oliveira, T. M., van Beek, L., Shilliday, F., Debreczeni, J., Phillips, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Cryo-EM:+The+resolution+revolution+and+drug+discovery.">Cryo-EM: The resolution revolution and drug discovery.</a> SLAS Discovery. 26 (1), 17-31 (2021).</li><li>Danev, R., Yanagisawa, H., Kikkawa, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Cryo-EM+performance+testing+of+hardware+and+data+acquisition+strategies.">Cryo-EM performance testing of hardware and data acquisition strategies.</a> Microscopy. 70 (6), 487-497 (2021).</li><li>Mastronarde, D. N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automated+electron+microscope+tomography+using+robust+prediction+of+specimen+movements.">Automated electron microscope tomography using robust prediction of specimen movements.</a> Journal of Structural Biology. 152 (1), 36-51 (2005).</li><li>. Tomography 5 and Tomo Live Software User-friendly batch acquisition for and on-the-fly reconstruction for cryo-electron tomography Datasheet Available from: <a target="_blank" href="https://assets.thermofisher.com/TFS-Assets/MSD/Datasheets/tomography-5-software-ds0362.pdf">https://assets.thermofisher.com/TFS-Assets/MSD/Datasheets/tomography-5-software-ds0362.pdf</a> (2022)</li><li>Danev, R., Baumeister, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Expanding+the+boundaries+of+cryo-EM+with+phase+plates.">Expanding the boundaries of cryo-EM with phase plates.</a> Current Opinion in Structural Biology. 46, 87-94 (2017).</li><li>Hylton, R. K., Swulius, M. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Challenges+and+triumphs+in+cryo-electron+tomography.">Challenges and triumphs in cryo-electron tomography.</a> iScience. 24 (9), (2021).</li><li>Turk, M., Baumeister, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+promise+and+the+challenges+of+cryo-electron+tomography.">The promise and the challenges of cryo-electron tomography.</a> FEBS Letters. 594 (20), 3243-3261 (2020).</li><li>Oikonomou, C. M., Jensen, G. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Cellular+electron+cryotomography:+Toward+structural+biology+in+situ.">Cellular electron cryotomography: Toward structural biology in situ.</a> Annual Review of Biochemistry. 86, 873-896 (2017).</li><li>Wagner, J., Schaffer, M., Fern&#225;ndez-Busnadiego, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Cryo-electron+tomography-the+cell+biology+that+came+in+from+the+cold.">Cryo-electron tomography-the cell biology that came in from the cold.</a> FEBS Letters. 591 (17), 2520-2533 (2017).</li><li>Lam, V., Villa, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Practical+approaches+for+Cryo-FIB+milling+and+applications+for+cellular+cryo-electron+tomography.">Practical approaches for Cryo-FIB milling and applications for cellular cryo-electron tomography.</a> Methods in Molecular Biology. 2215, 49-82 (2021).</li><li>Chreifi, G., Chen, S., Metskas, L. A., Kaplan, M., Jensen, G. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Rapid+tilt-series+acquisition+for+electron+cryotomography.">Rapid tilt-series acquisition for electron cryotomography.</a> Journal of Structural Biology. 205 (2), 163-169 (2019).</li><li>Eisenstein, F., Danev, R., Pilhofer, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Improved+applicability+and+robustness+of+fast+cryo-electron+tomography+data+acquisition.">Improved applicability and robustness of fast cryo-electron tomography data acquisition.</a> Journal of Structural Biology. 208 (2), 107-114 (2019).</li><li>Esteva, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Deep+learning-enabled+medical+computer+vision.">Deep learning-enabled medical computer vision.</a> npj Digital Medicine. 4 (1), (2021).</li><li>Liu, Y. -. T., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Isotropic+reconstruction+of+electron+tomograms+with+deep+learning.">Isotropic reconstruction of electron tomograms with deep learning.</a> bioRxiv. , (2021).</li><li>Moebel, E., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Deep+learning+improves+macromolecule+identification+in+3D+cellular+cryo-electron+tomograms.">Deep learning improves macromolecule identification in 3D cellular cryo-electron tomograms.</a> Nature Methods. 18 (11), 1386-1394 (2021).</li><li>Chen, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Convolutional+neural+networks+for+automated+annotation+of+cellular+cryo-electron+tomograms.">Convolutional neural networks for automated annotation of cellular cryo-electron tomograms.</a> Nature Methods. 14 (10), 983-985 (2017).</li><li>Ronneberger, O., Fischer, P., Brox, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=U-net:+Convolutional+networks+for+biomedical+image+segmentation.">U-net: Convolutional networks for biomedical image segmentation.</a> Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 9351, 234-241 (2015).</li><li>Kremer, J. R., Mastronarde, D. N., McIntosh, J. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Computer+visualization+of+three-dimensional+image+data+using+IMOD.">Computer visualization of three-dimensional image data using IMOD.</a> Journal of Structural Biology. 116 (1), 71-76 (1996).</li><li>Iancu, C. V., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+&amp;#34;flip-flop&amp;#34;+rotation+stage+for+routine+dual-axis+electron+cryotomography.">A &amp;#34;flip-flop&amp;#34; rotation stage for routine dual-axis electron cryotomography.</a> Journal of Structural Biology. 151 (3), 288-297 (2005).</li></ol>

The open-access license for this protocol was paid for by Object Research Systems.

This protocol lays out a procedure for using Dragonfly 2022.1 software to train a multi-class U-Net from a single tomogram, and how to infer that network to other tomograms that do not need to be from the same dataset. Training is relatively quick (can be as fast as 3-5 min per epoch or as slow as a few hours, depending entirely on the network that is being trained and the hardware used), and retraining a network to improve its learning is intuitive. As long as the preprocessing steps are carried out for every tomogram, inference is typically robust.
Consistent preprocessing is the most critical step for deep learning inference. There are many imaging filters in the software and the user can experiment to determine which filters work best for particular datasets; note that whatever filtering is used on the training tomogram must be applied in the same way to the inference tomograms. Care must also be taken to provide the network with accurate and sufficient training information. It is vital that all features segmented within the training slices are segmented as carefully and precisely as possible.
Image segmentation is facilitated by a sophisticated commercial-grade user interface. It provides all the necessary tools for hand segmentation and allows for the simple reassignment of voxels from any one class into another prior to training and retraining. The user is allowed to hand-segment voxels within the whole context of the tomogram, and they are given multiple views and the ability to rotate the volume freely. Additionally, the software provides the ability to use multi-class networks, which tend to perform better16 and are faster than segmenting with multiple single-class networks.
There are, of course, limitations to a neural network&#39;s capabilities. Cryo-ET data are, by nature, very noisy and limited in angular sampling, which leads to orientation-specific distortions in identical objects21. Training relies on an expert to hand-segment structures accurately, and a successful network is only as good (or as bad) as the training data it is given. Image filtering to boost signal is helpful for the trainer, but there are still many cases where accurately identifying all pixels of a given structure is difficult. It is, therefore, important that great care is taken when creating the training segmentation so that the network has the best information possible to learn during training.
This workflow can be easily modified for each user&#39;s preference. While it is essential that all tomograms be preprocessed in exactly the same manner, it is not necessary to use the exact filters used in the protocol. The software has numerous image filtering options, and it is recommended to optimize these for the user&#39;s particular data before setting out on a large segmentation project spanning many tomograms. There are also quite a few network architectures available to use: a multi-slice U-Net has been found to work best for the data from this lab, but another user might find that another architecture (such as a 3D U-Net or a Sensor 3D) works better. The segmentation wizard provides a convenient interface for comparing the performance of multiple networks using the same training data.
Tools like the ones presented here will make hand segmentation of full tomograms a task of the past. With well-trained neural networks that are robustly inferable, it is entirely feasible to create a workflow where tomographic data is reconstructed, processed, and fully segmented as quickly as the microscope can collect it.

Hardware and software developments in the past decade have resulted in a &#34;resolution revolution&#34; for cryo-electron microscopy (cryo-EM)1,2. With better and faster detectors3, software to automate data collection4,5, and signal boosting advances such as phase plates6, collecting large amounts of high-resolution cryo-EM data is relatively straightforward.
Cryo-ET delivers unprecedented insight into cellular ultrastructure in a native, hydrated state7,8,9,10. The primary limitation is sample thickness, but with the adoption of methods such as focused ion beam (FIB) milling, where thick cellular and tissue samples are thinned for tomography11, the horizon for what can be imaged with cryo-ET is constantly expanding. The newest microscopes are capable of producing well over 50 tomograms a day, and this rate is only projected to increase due to the development of rapid data collection schemes12,13. Analyzing the vast amounts of data produced by cryo-ET remains a bottleneck for this imaging modality.
Quantitative analysis of tomographic information requires that it first be annotated. Traditionally, this requires hand segmentation by an expert, which is time-consuming; depending on the molecular complexity contained within the cryo-tomogram, it can take hours to days of dedicated attention. Artificial neural networks are an appealing solution to this problem since they can be trained to do the bulk of the segmentation work in a fraction of the time. Convolutional neural networks (CNNs) are especially suited to computer vision tasks14 and have recently been adapted for the analysis of cryo-electron tomograms15,16,17.
Traditional CNNs require many thousands of annotated training samples, which is not often possible for biological image analysis tasks. Hence, the U-Net architecture has excelled in this space18 because it relies on data augmentation to successfully train the network, minimizing the dependency on large training sets. For instance, a U-Net architecture can be trained with only a few slices of a single tomogram (four or five slices) and robustly inferred to other tomograms without retraining. This protocol provides a step-by-step guide for training U-Net neural network architectures to segment electron cryo-tomograms within Dragonfly 2022.119.
Dragonfly is commercially developed software used for 3D image segmentation and analysis by deep learning models, and it is freely available for academic use (some geographical restrictions apply). It has an advanced graphical interface that allows a non-expert to take full advantage of the powers of deep learning for both semantic segmentation and image denoising. This protocol demonstrates how to preprocess and annotate&#160;cryo-electron tomograms within Dragonfly for training artificial neural networks, which can then be inferred to rapidly segment large datasets. It further discusses and briefly demonstrates how to use segmented data for further analysis such as filament tracing and coordinate extraction for sub-tomogram averaging.

<table><tbody><tr><td>Dragonfly 2022.1</td><td>Object Research Systems</td><td></td><td>https://www.theobjects.com/dragonfly/index.html</td></tr><tr><td>E18 Rat Dissociated Hippocampus</td><td>Transnetyx Tissue</td><td>KTSDEDHP</td><td>https://tissue.transnetyx.com/faqs</td></tr><tr><td>IMOD</td><td>University of Colorado</td><td></td><td>https://bio3d.colorado.edu/imod/</td></tr><tr><td>Intel&amp;reg; Xeon&amp;reg; Gold 6124 CPU 3.2GHz</td><td>Intel</td><td></td><td>https://www.intel.com/content/www/us/en/products/sku/120493/intel-xeon-gold-6134-processor-24-75m-cache-3-20-ghz/specifications.html</td></tr><tr><td>NVIDIA Quadro P4000</td><td>NVIDIA</td><td></td><td>https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/productspage/quadro/quadro-desktop/quadro-pascal-p4000-data-sheet-a4-nvidia-704358-r2-web.pdf</td></tr><tr><td>Windows 10 Enterprise 2016</td><td>Microsoft</td><td></td><td>https://www.microsoft.com/en-us/evalcenter/evaluate-windows-10-enterprise</td></tr><tr><td>Workstation Minimum Requirements</td><td></td><td></td><td>https://theobjects.com/dragonfly/system-requirements.html</td></tr></tbody></table>

deep learning-based segmentation of cryo-electron tomograms

Cryo-electron tomography (cryo-ET) allows researchers to image cells in their native, hydrated state at the highest resolution currently possible. The technique has several limitations, however, that make analyzing the data it generates time-intensive and difficult. Hand segmenting a single tomogram can take from hours to days, but a microscope can easily generate 50 or more tomograms a day. Current deep learning segmentation programs for cryo-ET do exist, but are limited to segmenting one structure at a time. Here, multi-slice U-Net convolutional neural networks are trained and applied to automatically segment multiple structures simultaneously within cryo-tomograms. With proper preprocessing, these networks can be robustly inferred to many tomograms without the need for training individual networks for each tomogram. This workflow dramatically improves the speed with which cryo-electron tomograms can be analyzed by cutting segmentation time down to under 30 min in most cases. Further, segmentations can be used to improve the accuracy of filament tracing within a cellular context and to rapidly extract coordinates for subtomogram averaging.

Following the protocol, a five-slice U-Net was trained on a single tomogram (Figure 2A) to identify five classes: Membrane, Microtubules, Actin, Fiducial markers, and Background. The network was iteratively trained a total of three times, and then applied to the tomogram to fully segment and annotate it (Figure 2B,C). Minimal cleanup was performed using steps 7.1 and 7.2. The next three tomograms of interest (Figure 2D,G,J) were loaded into the software for preprocessing. Prior to image import, one of the tomograms (Figure 2J) required pixel size adjustment from 17.22 &#197;/px to 13.3 &#197;/px as it was collected on a different microscope at a slightly different magnification. The IMOD program squeezevol was used for resizing with the following command:
&#39;squeezevol -f 0.772 inputfile.mrc outputfile.mrc&#39;
In this command, -f refers to the factor by which to alter the pixel size (in this case: 13.3/17.22). After import, all three inference targets were preprocessed according to steps 3.2 and 3.3, and then the five-slice U-Net was applied. Minimal cleanup was again performed. The final segmentations are displayed in Figure 2.
Microtubule segmentations from each tomogram were exported as binary (step 7.4) TIF files, converted to MRC (IMOD tif2mrc program), and then used for cylinder correlation and filament tracing. Binary segmentations of filaments result in much more robust filament tracing than tracing over tomograms. Coordinate maps from filament tracing (Figure 3) will be used for further analysis, such as nearest neighbor measurements (filament packing) and helical sub-tomogram averaging along single filaments to determine microtubule orientation.
Unsuccessful or inadequately trained networks are easy to determine. A failed network will be unable to segment any structures at all, whereas an inadequately trained network typically will segment some structures correctly and have a significant number of false positives and false negatives. These networks can be corrected and iteratively trained to improve their performance. The segmentation wizard automatically calculates a model&#39;s Dice similarity coefficient (called score in the SegWiz) after it is trained. This statistic gives an estimate of the similarity between the training data and the U-Net segmentation. Dragonfly 2022.1 also has a built-in tool to evaluate a model&#39;s performance that can be accessed in the Artificial Intelligence tab at the top of the interface (see documentation for usage).
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/64435/64435fig02.jpg" /> 
Figure 2: Inference. (A-C) Original training tomogram of a DIV 5 hippocampal rat neuron, collected in 2019 on a Titan Krios. This is a backprojected reconstruction with CTF correction in IMOD. (A) The yellow box represents the region where hand segmentation was performed for training input. (B) 2D segmentation from the U-Net after training is complete. (C) 3D rendering of the segmented regions showing membrane (blue), microtubules (green), and actin (red). (D-F) DIV 5 hippocampal rat neuron from the same session as the training tomogram. (E) 2D segmentation from the U-Net with no additional training and quick cleanup. Membrane (blue), microtubules (green), actin (red), fiducials (pink). (F) 3D rendering of the segmented regions. (G-I)&#160;DIV 5 hippocampal rat neuron from the 2019 session. (H) 2D segmentation from the U-Net with quick cleanup and (I) 3D rendering. (J-L) DIV 5 hippocampal rat neuron, collected in 2021 on a different Titan Krios at a different magnification. Pixel size has been changed with the IMOD program squeezevol to match the training tomogram. (K) 2D segmentation from the U-Net with quick cleanup, demonstrating robust inference across datasets with proper preprocessing and (L) 3D rendering of segmentation. Scale bars = 100 nm. Abbreviations: DIV = days in vitro; CTF = contrast transfer function. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/64435/64435fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/64435/64435fig03.jpg" /> 
Figure 3: Filament tracing improvement. (A) Tomogram of a DIV 4 rat hippocampal neuron, collected on a Titan Krios. (B)&#160;Correlation map generated from cylinder correlation over actin filaments. (C)&#160;Filament tracing of actin using the intensities of the actin filaments in the correlation map to define parameters. Tracing captures the membrane and microtubules, as well as noise, while trying to trace just actin. (D)&#160;U-Net segmentation of tomogram. Membrane highlighted in blue, microtubules in red, ribosomes in orange, triC in purple, and actin in green. (E)&#160;Actin segmentation extracted as a binary mask for filament tracing. (F)&#160;Correlation map generated from cylinder correlation with the same parameters from (B). (G) Significantly improved filament tracing of just actin filaments from the tomogram. Abbreviation: DIV = days in vitro. <a href="https://www-jove-com.remotexs.ntu.edu.sg/files/ftp_upload/64435/64435fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
Supplemental File 1: The tomogram used in this protocol and the multi-ROI that was generated as training input are included as a bundled dataset (Training.ORSObject). See&#160;https://datadryad.org/stash/dataset/doi:10.5061/dryad.rxwdbrvct.

Watch this Scientific Journal Video about Deep Learning-Based Segmentation of Cryo-Electron Tomograms at JoVE.com

Deep Learning-Based Segmentation of Cryo-Electron Tomograms

This is a method for training a multi-slice U-Net for multi-class segmentation of cryo-electron tomograms using a portion of one tomogram as a training input. We describe how to infer this network to other tomograms and how to extract segmentations for further analyses, such as subtomogram averaging and filament tracing.

Cryo-electron tomography (cryo-ET) allows researchers to image cells in their native, hydrated state at the highest resolution currently possible. The ...

deep-learning-based-segmentation-of-cryo-electron-tomograms

Nanyang Technological University

Research

JoVE Journal

Biology

8.3K Views.  Pennsylvania State University-College of Medicine. This is a method for training a multi-slice U-Net for multi-class segmentation of cryo-electron tomograms using a portion of one tomogram as a training input. We describe how to infer this network to other tomograms and how to extract segmentations for further analyses, such as subtomogram averaging and filament tracing.

Video: Deep Learning-Based Segmentation of Cryo-Electron Tomograms

From Voxels to Knowledge: A Practical Guide to the Segmentation of Complex Electron Microscopy 3D-Data

Modern 3D electron microscopy approaches have recently allowed unprecedented insight into the 3D ultrastructural organization of cells and tissues, enabling the visualization of large macromolecular machines, such as adhesion complexes, as well as higher-order structures, such as the cytoskeleton and cellular organelles in their respective cell and tissue context. Given the inherent complexity of cellular volumes, it is essential to first extract the features of interest in order to allow visualization, quantification, and therefore comprehension of their 3D organization. Each data set is defined by distinct characteristics, e.g., signal-to-noise ratio, crispness (sharpness) of the data, heterogeneity of its features, crowdedness of features, presence or absence of characteristic shapes that allow for easy identification, and the percentage of the entire volume that a specific region of interest occupies. All these characteristics need to be considered when deciding on which approach to take for segmentation.
The six different 3D ultrastructural data sets presented were obtained by three different imaging approaches: resin embedded stained electron tomography, focused ion beam- and serial block face- scanning electron microscopy (FIB-SEM, SBF-SEM) of mildly stained and heavily stained samples, respectively. For these data sets, four different segmentation approaches have been applied: (1) fully manual model building followed solely by visualization of the model, (2) manual tracing segmentation of the data followed by surface rendering, (3) semi-automated approaches followed by surface rendering, or (4) automated custom-designed segmentation algorithms followed by surface rendering and quantitative analysis. Depending on the combination of data set characteristics, it was found that typically one of these four categorical approaches outperforms the others, but depending on the exact sequence of criteria, more than one approach may be successful. Based on these data, we propose a triage scheme that categorizes both objective data set characteristics and subjective personal criteria for the analysis of the different data sets.

The bottleneck for cellular 3D electron microscopy is feature extraction (segmentation) in highly complex 3D density maps. We have developed a set of criteria, which provides guidance regarding which segmentation approach (manual, semi-automated, or automated) is best suited for different data types, thus providing a starting point for effective segmentation.

Modern 3D electron microscopy approaches have recently allowed unprecedented insight into the 3D ultrastructural organization of cells and tissues, ...

Bioengineering

In vivo Imaging of Deep Cortical Layers using a Microprism

We present a protocol for in vivo imaging of cortical tissue using a deep-brain imaging probe in the shape of a microprism.  Microprisms are 1-mm in size and have a reflective coating on the hypotenuse to allow internal reflection of excitation and emission light.  The microprism probe simultaneously images multiple cortical layers with a perspective typically seen only in slice preparations.  Images are collected with a large field-of-view (~900 &mu;m).  In addition, we provide details on the non-survival surgical procedure and microscope setup.  Representative results include images of layer V pyramidal neurons from Thy-1 YFP-H mice showing their apical dendrites extending through the superficial cortical layer and extending into tufts.  Resolution was sufficient to image dendritic spines near the soma of layer V neurons.  A tail-vein injection of fluorescent dye reveals the intricate network of blood vessels in the cortex.  Line-scanning of red blood cells (RBCs) flowing through the capillaries reveals RBC velocity and flux rates can be obtained. This novel microprism probe is an elegant, yet powerful new method of visualizing deep cellular structures and cortical function in vivo.

In vivo Imaging of Deep Cortical Layers using a Microprism

Right-angle microprisms inserted into the mouse neocortex allows for deep imaging of multiple cortical layers with a viewpoint typically found in slice. One-millimeter microprisms offer a wide field-of-view (~900 &mu;m) and spatial resolutions sufficient to resolve dendritic spines. We demonstrate layer V neuronal imaging and neocortical vascular imaging using microprisms.

We present a protocol for in vivo imaging of cortical tissue using a deep-brain imaging probe in the shape of a microprism.  Microprisms are 1-mm in size ...

Preparation of Primary Neurons for Visualizing Neurites in a Frozen-hydrated State Using Cryo-Electron Tomography

Neurites, both dendrites and axons, are neuronal cellular processes that enable the conduction of electrical impulses between neurons. Defining the structure of neurites is critical to understanding how these processes move materials and signals that support synaptic communication. Electron microscopy (EM) has been traditionally used to assess the ultrastructural features within neurites; however, the exposure to organic solvent during dehydration and resin embedding can distort structures. An important unmet goal is the formulation of procedures that allow for structural evaluations not impacted by such artifacts.
Here, we have established a detailed and reproducible protocol for growing and flash-freezing whole neurites of different primary neurons on electron microscopy grids followed by their examination with cryo-electron tomography (cryo-ET). This technique allows for 3-D visualization of frozen, hydrated neurites at nanometer resolution, facilitating assessment of their morphological differences. Our protocol yields an unprecedented view of dorsal root ganglion (DRG) neurites, and a visualization of hippocampal neurites in their near-native state. As such, these methods create a foundation for future studies on neurites of both normal neurons and those impacted by neurological disorders.

To preserve neuronal processes for ultrastructural analysis, we describe a protocol for plating of primary neurons on electron microscopy grids followed by flash freezing, yielding samples suspended in a layer of vitreous ice. These samples can be examined with a cryo-electron microscope to visualize structures at the nanometer scale.

Neurites, both dendrites and axons, are neuronal cellular processes that enable the conduction of electrical impulses between neurons. Defining the ...

Neuroscience

Using Tomoauto: A Protocol for High-throughput Automated Cryo-electron Tomography

Cryo-electron tomography (Cryo-ET) is a powerful three-dimensional (3-D) imaging technique for visualizing macromolecular complexes in their native context at a molecular level. The technique involves initially preserving the sample in its native state by rapidly freezing the specimen in vitreous ice, then collecting a series of micrographs from different angles at high magnification, and finally computationally reconstructing a 3-D density map. The frozen-hydrated specimen is extremely sensitive to the electron beam and so micrographs are collected at very low electron doses to limit the radiation damage. As a result, the raw cryo-tomogram has a very low signal to noise ratio characterized by an intrinsically noisy image. To better visualize subjects of interest, conventional imaging analysis and sub-tomogram averaging in which sub-tomograms of the subject are extracted from the initial tomogram and aligned and averaged are utilized to improve both contrast and resolution. Large datasets of tilt-series are essential to understanding and resolving the complexes at different states, conditions, or mutations as well as obtaining a large enough collection of sub-tomograms for averaging and classification. Collecting and processing this data can be a major obstacle preventing further analysis. Here we describe a high-throughput cryo-ET protocol based on a computer-controlled 300kV cryo-electron microscope, a direct detection device (DDD) camera and a highly effective, semi-automated image-processing pipeline software wrapper library tomoauto developed in-house. This protocol has been effectively utilized to visualize the intact type III secretion system (T3SS) in Shigella flexneri minicells. It can be applicable to any project suitable for cryo-ET.

We present a protocol on how to utilize high-throughput cryo-electron tomography to determine high resolution in situ structures of molecular machines. The protocol permits large amounts of data to be processed, avoids common bottlenecks and reduces resource downtime, allowing the user to focus on important biological questions.

Cryo-electron tomography (Cryo-ET) is a powerful three-dimensional (3-D) imaging technique for visualizing macromolecular complexes in their native ...

Preparing Lamellae from Vitreous Biological Samples Using a Dual-Beam Scanning Electron Microscope for Cryo-Electron Tomography

Presented here is a protocol for preparing cryo-lamellae from plunge-frozen grids of Plasmodium falciparum-infected&#160;human erythrocytes, which could easily be adapted for other biological samples. The basic principles for preparing samples, milling, and viewing lamellae are common to all instruments and the protocol can be followed as a general guide to on-grid cryo-lamella preparation for cryo-electron microscopy (cryoEM) and cryo-electron tomography (cryoET). Electron microscopy grids supporting the cells are plunge-frozen into liquid nitrogen-cooled liquid ethane using a manual or automated plunge freezer, then screened on a light microscope equipped with a cryo-stage. Frozen grids are transferred into a cryo-scanning electron microscope equipped with a focused ion beam (cryoFIB-SEM). Grids are routinely sputter coated prior to milling, which aids dispersal of charge build-up during milling. Alternatively, an e-beam rotary coater can be used to apply a layer of carbon-platinum to the grids, the exact thickness of which can be more precisely controlled. Once inside the cryoFIB-SEM an additional coating of an organoplatinum compound is applied to the surface of the grid via a gas injection system (GIS). This layer protects the front edge of the lamella as it is milled, the integrity of which is critical for achieving uniformly thin lamellae. Regions of interest are identified via SEM and milling is carried out in a step-wise fashion, reducing the current of the ion beam as the lamella reaches electron transparency, in order to avoid excessive heat generation. A grid with multiple lamellae is then transferred to a transmission electron microscope (TEM) under cryogenic conditions for tilt-series acquisition. A robust and contamination-free workflow for lamella preparation is an essential step for downstream techniques, including cellular cryoEM, cryoET, and sub-tomogram averaging. Development of these techniques, especially for lift-out and milling of high-pressure frozen samples, is of high-priority in the field.

Using focused ion beam milling to produce vitreous on-grid lamellae from plunge frozen biological samples for cryo-electron tomography.

Presented here is a protocol for preparing cryo-lamellae from plunge-frozen grids of Plasmodium falciparum-infected&#160;human erythrocytes, which could ...

Biochemistry

Simplified Volumetric Models as an Effective Strategy for Segmenting Actin Networks in Cryo-Electron Tomograms

Efficient methods for the extraction of features of interest remain one of the biggest challenges for the interpretation of cryo-electron tomograms. Various automated approaches have been proposed, many of which work well for high-contrast datasets where the features of interest can be easily detected and are clearly separated from one another. Our inner ear stereocilia cryo-electron tomographic datasets are characterized by a dense array of hexagonally packed actin filaments that are frequently cross-connected. These features make automated segmentation very challenging, further aggravated by the high-noise environment of cryo-electron tomograms and the high complexity of the densely packed features. Using prior knowledge about the actin bundle organization, we have placed layers of a highly simplified ball-and-stick actin model to first obtain a global fit to the density map, followed by regional and local adjustments of the model. We show that volumetric model building not only allows us to deal with the high complexity, but also provides precise measurements and statistics about the actin bundle. Volumetric models also serve as anchoring points for local segmentation, such as in the case of the actin-actin cross connectors. Volumetric model building, particularly when further augmented by computer-based automated fitting approaches, can be a powerful alternative when conventional automated segmentation approaches are not successful.

Here, we present a protocol to place simplified volumetric models into noisy, complex, tomographic 3D volumes. This allows the fast segmentation of actin filament densities, detection of systematic filament bending and of gaps in hair bundle filaments, as well as convenient quantification of volumetric model properties, such as distances.

Efficient methods for the extraction of features of interest remain one of the biggest challenges for the interpretation of cryo-electron tomograms. ...

Strategies for Optimization of Cryogenic Electron Tomography Data Acquisition

Cryogenic electron tomography (cryoET) is a powerful method to study the 3D structure of biological samples in a close-to-native state. Current state-of-the-art cryoET combined with subtomogram averaging analysis enables the high-resolution structural determination of macromolecular complexes that are present in multiple copies within tomographic reconstructions. Tomographic experiments usually require a vast amount of tilt series to be acquired by means of high-end transmission electron microscopes with important operational running-costs. Although the throughput and reliability of automated data acquisition routines have constantly improved over the recent years, the process of selecting regions of interest at which a tilt series will be acquired cannot be easily automated and it still relies on the user's manual input. Therefore, the set-up of a large-scale data collection session is a time-consuming procedure that can considerably reduce the remaining microscope time available for tilt series acquisition. Here, the protocol describes the recently developed implementations based on the SerialEM package and the PyEM software that significantly improve the time-efficiency of grid screening and large-scale tilt series data collection. The presented protocol illustrates how to use SerialEM scripting functionalities to fully automate grid mapping, grid square mapping, and tilt series acquisition. Furthermore, the protocol describes how to use PyEM to select additional acquisition targets in off-line mode after automated data collection is initiated. To illustrate this protocol, its application in the context of high-end data collection of Sars-Cov-2 tilt series is described. The presented pipeline is particularly suited to maximizing the time-efficiency of tomography experiments that require a careful selection of acquisition targets and at the same time a large amount of tilt series to be collected.

The increasing demand for large-scale data collection in cryogenic electron tomography requires high-throughput image acquisition routines. Described here is a protocol that implements the recent developments of advanced acquisition strategies aimed at maximizing the time-efficiency and throughput of tomographic data collection.

Cryogenic electron tomography (cryoET) is a powerful method to study the 3D structure of biological samples in a close-to-native state. Current ...

Cryo-Electron Tomography Remote Data Collection and Subtomogram Averaging

Cryo-electron tomography (cryo-ET) has been gaining momentum in recent years, especially since the introduction of direct electron detectors, improved automated acquisition strategies, preparative techniques that expand the possibilities of what the electron microscope can image at high-resolution using cryo-ET and new subtomogram averaging software. Additionally, data acquisition has become increasingly streamlined, making it more accessible to many users. The SARS-CoV-2 pandemic has further accelerated remote cryo-electron microscopy (cryo-EM) data collection, especially for single-particle cryo-EM, in many facilities globally, providing uninterrupted user access to state-of-the-art instruments during the pandemic. With the recent advances in Tomo5 (software for 3D electron tomography), remote cryo-ET data collection has become robust and easy to handle from anywhere in the world. This article aims to provide a detailed walk-through, starting from the data collection setup in the tomography software for the process of a (remote) cryo-ET data collection session with detailed troubleshooting. The (remote) data collection protocol is further complemented with the workflow for structure determination at near-atomic resolution by subtomogram averaging with emClarity, using apoferritin as an example.

The present protocol describes high-resolution cryo-electron tomography remote data acquisition using Tomo5 and subsequent data processing and subtomogram averaging using emClarity. Apoferritin is used as an example to illustrate detailed step-by-step processes to achieve a cryo-ET structure at 2.86 &#197; resolution.

Cryo-electron tomography (cryo-ET) has been gaining momentum in recent years, especially since the introduction of direct electron detectors, improved ...

Visualization of Organelles In Situ by Cryo-STEM Tomography

Cryogenic electron microscopy (cryo-EM) relies on the imaging of biological or organic specimens embedded in their native aqueous medium; water is solidified into a glass (i.e., vitrified) without crystallization. The cryo-EM method is widely used to determine the structure of biological macromolecules recently at a near-atomic resolution. The approach has been extended to the study of organelles and cells using tomography, but the conventional mode of wide-field transmission EM imaging suffers a severe limitation in the specimen thickness. This has led to a practice of milling thin lamellae using a focused ion beam; the high resolution is obtained by subtomogram averaging from the reconstructions, but three-dimensional relations outside the remaining layer are lost. The thickness limitation can be circumvented by scanned probe imaging, similar to the scanning EM or the confocal laser scanning microscope. While scanning transmission electron microscopy (STEM) in materials science provides atomic resolution in single images, the sensitivity of cryogenic biological specimens to electron irradiation requires special considerations. This protocol presents a setup for cryo-tomography using STEM. The basic topical configuration of the microscope is described for both two- and three-condenser systems, while automation is provided by the non-commercial SerialEM software. Enhancements for batch acquisition and correlative alignment to previously-acquired fluorescence maps are also described. As an example, we show the reconstruction of a mitochondrion, pointing out the inner and outer membrane and calcium phosphate granules, as well as surrounding microtubules, actin filaments, and ribosomes. Cryo-STEM tomography excels in revealing the theater of organelles in the cytoplasm and, in some cases, even the nuclear periphery of adherent cells in culture.

Visualization of Organelles In Situ by Cryo-STEM Tomography

Cryo-STEM tomography provides a means to visualize organelles of intact cells without embedding, sectioning, or other invasive preparations. The 3D resolution obtained is currently in the range of a few nanometers, with a field of view of several micrometers and an accessible thickness in the order of 1 &#181;m.

Cryogenic electron microscopy (cryo-EM) relies on the imaging of biological or organic specimens embedded in their native aqueous medium; water is ...

VR Interaction Tools: 3D Cryo-ET Data Segmentation

Leveraging Virtual Reality for Immersive Segmentation and Analysis of Cryo-Electron Tomography Data

Cryo-electron tomography (cryo-ET) is a powerful technique for visualizing the ultrastructure of cells in three dimensions (3D) at nanometer resolution. However, the manual segmentation of cellular components in cryo-ET data remains a significant bottleneck due to its complexity and time-consuming nature. In this work, we present a novel segmentation workflow that integrates advanced virtual reality (VR) software to enhance both the efficiency and accuracy of segmenting cryo-ET datasets. This workflow leverages an immersive VR tool with intuitive 3D interaction, enabling users to navigate and annotate complex cellular structures in a more natural and interactive environment. To evaluate the effectiveness of the workflow, we applied it to the segmentation of mitochondria in retinal pigment epithelium (RPE1) cells. Mitochondria, essential for cellular energy production and signaling, exhibit dynamic morphological changes, making them an ideal test sample. The VR software facilitated precise delineation of mitochondrial membranes and internal structures, enabling downstream analysis of the segmented membrane structures. We demonstrate that this VR-based segmentation workflow significantly improves the user experience while maintaining accurate segmentation of intricate cellular structures in cryo-ET data. This approach holds promise for broad applications in structural cell biology and science education, offering a transformative tool for researchers engaged in detailed cellular analysis.

Cryo-electron tomography (cryo-ET) enables 3D visualization of cellular ultrastructure at nanometer resolution, but manual segmentation remains time-consuming and complex. We present a novel workflow that integrates advanced virtual reality software for segmenting cryo-ET tomograms, showcasing its effectiveness through the segmentation of mitochondria in mammalian cells.

Cryo-electron tomography (cryo-ET) is a powerful technique for visualizing the ultrastructure of cells in three dimensions (3D) at nanometer resolution. ...

Deep Learning-Based Segmentation of Cryo-Electron Tomograms

Summary

Explore More Videos

Chapters in this video

From Voxels to Knowledge: A Practical Guide to the Segmentation of Complex Electron Microscopy 3D-Data

In vivo Imaging of Deep Cortical Layers using a Microprism

Preparation of Primary Neurons for Visualizing Neurites in a Frozen-hydrated State Using Cryo-Electron Tomography

Using Tomoauto: A Protocol for High-throughput Automated Cryo-electron Tomography

Preparing Lamellae from Vitreous Biological Samples Using a Dual-Beam Scanning Electron Microscope for Cryo-Electron Tomography

Simplified Volumetric Models as an Effective Strategy for Segmenting Actin Networks in Cryo-Electron Tomograms

Strategies for Optimization of Cryogenic Electron Tomography Data Acquisition

Cryo-Electron Tomography Remote Data Collection and Subtomogram Averaging

Visualization of Organelles In Situ by Cryo-STEM Tomography

Leveraging Virtual Reality for Immersive Segmentation and Analysis of Cryo-Electron Tomography Data