Big Data Electron and Correlative Microscopy from Instrument to publication
June 21, 2023
Big Data Electron and Correlative Microscopy from Instrument to publication, is one of the 3 specialised programs of the Australian Characterisation Commons at Scale (ACCS) Project.
This program addresses the challenges generated by new EM and light techniques, this includes cryo-electron microscopy (CryoEM) which is becoming broadly available across Australia, and new material techniques.
This specialised program deployed the EM Data Processing Portal (EMDPP), a tool that provides access to high-performance GPU clusters for processing EM data using CryoSPARC and LiberTEM services.
The Portal consists of 2 nodes: one GPU cluster at QCIF (Queensland Cyber Infrastructure Foundation) and another GPU cluster at Monash University. This service has been heavily promoted across EM facilities, ARDC and conferences, with multiple news articles, social media and newsletters inclusions. The EMDPP is one of the greatest outcomes of the project and the team is working towards its sustainability beyond the project.
Particle Imaging depoT using Storage caCHing Infrastructure (PITSCHI): A data management tool Based on the data management framework Clowder. Pitschi has been designed as part of the ACCS WP4 for the Centre of Microscopy and Microanalysis at The University of Queensland as an end-to-end solution that supports the entire research data lifecycle by storing, indexing and annotating data generated at the centre.
Pitschi's team's aim is to deploy this tool at other universities and research facilities. Currently, a training and information session is delivered weekly to assist researchers with the data generated at CMM and how to use PITSCHI in their workflow.
Outcomes:
WP4.0
Poger, D., van Schyndel, J., Nguyen, H., Silver, J., & Goscinski, W. J. (2021). Orchestration and management of data generated by big-data electron microscopy instruments: A Discovery report. Zenodo. http://doi.org/10.5281/zenodo.4744876
Poger, D., Yen, L., & Braet, F. (2023). Big data in contemporary electron microscopy: challenges and opportunities in data transfer, compute and management. Histochemistry and Cell Biology, Advance online publication. https://doi.org/10.1007/s00418-023-02191-8
WP4.1
van Schyndel, J., Silver, J., & Poger, D. (2021). Investigation, prototyping and deployment of a higher-level service for data transport. Zenodo. https://doi.org/10.5281/zenodo.5124450
WP4.2
Silver, J. (2023). Optimisation and automation of workflows from data capture to data processing in electron microscopy. Zenodo. https://doi.org/10.5281/zenodo.7039561
WP4.3
Myers, C., & Poger, D. (2022). Reproducible baselining and benchmarking of network performance with perfSONAR. Zenodo. https://doi.org/10.5281/zenodo.7018190
WP4.4
Silver, J. (2023). Real-time processing of microscopy data. Zenodo. https://doi.org/10.5281/zenodo.7039664
WP4.5
Nguyen, H. A., & White, A. (2023). Automation of particle counting and size characterisation in microscopy images. Zenodo. https://doi.org/10.5281/zenodo.7686749
WP4.7
van Schyndel, J. (2022). The Electron Microscopy Data Processing Portal: A new capability for data-intensive research. Zenodo. https://doi.org/10.5281/zenodo.7302408
WP4.9
Nguyen, H. A. (2023). Pitschi: a Clowder-based end-to-end data management tool. Zenodo. https://doi.org/10.5281/zenodo.7183431
WP4.10
Poger, D. (2023). Research data retention and disposal: Overview and guidelines. Zenodo. https://doi.org/10.5281/zenodo.7964322
WP4.11
Poger, D. (2023). Metadata collection at microscopy research facilities: Overview and recommendations. Zenodo. https://doi.org/10.5281/zenodo.7949932