Collaborating to solve large-scale research data transfer challenges
June 28, 2021
Electron microscopy (EM) and light techniques, including cryo-electron microscopy (CryoEM), are widely used for characterisation. Above is a A preliminary 3D cryo-EM structure of a bacteriophage (bacterial virus) protein reconstructed at Molecular Horizons. Source: University of Wollongong.
AARNet is working with the Australian Characterisation Commons at Scale (ACCS) project to address some of the big data transfer challenges faced by researchers using characterisation techniques and imaging collections, and scientists running instruments.
Characterisation is the process of measuring the properties of materials and is used across research in the natural, life and biomedical sciences and engineering. Electron microscopy (EM) and light techniques, including cryo-electron microscopy (CryoEM), are widely used for characterisation. The huge volumes of data generated need to be moved from instruments to and from data storage and high-performance computing (HPC) facilities, with these endpoints often located long distances apart.
A complex problem
Packaging up huge volumes of data so that they can be transferred between these locations across AARNet, the high-speed research network that connects them, is not always a simple task, and researchers sometimes resort to the risky practice of driving and flying hard drives around. Think of AARNet as a data superhighway, purpose built with the capacity to efficiently move large-scale research data at speed, but the data needs a vehicle to travel on the network, and the available vehicles don’t always perform as expected, particularly over long distances.
This is the complex problem that AARNet and ACCS are working together to address, with the help of Globus, a fast, reliable and secure data management service used by researchers at hundreds of universities and research institutions worldwide.
The work forms part of the ACCS project’s Characterisation Commons development. The ACCS project is supported by the Australian Research Data Commons and funded through the National Collaborative Research Infrastructure Strategy. The goal is to deliver a coherent and accessible informatics ecosystem that streamlines the data management and publishing pipelines, and resolves many of the big data challenges generated by new EM and light techniques.
Unifying access to systems
AARNet and ACCS have successfully trialled Globus for transferring data from powerful biological cryogenic electron microscopes located at the University of Wollongong (UOW)’s Molecular Horizons facility over AARNet’s high-speed network and into the Multi-modal Australian ScienceS Imaging and Visualisation Environment (MASSIVE), Australia’s specialised HPC research facility located at Monash University in Melbourne.
These atomic resolution microscopes with ultra-sensitive cameras that can detect single electrons are used by researchers for reconstructing 3D images via CryoEM, showing the structures of molecules at atomic detail. HPC is required for intensive processing and analysis of the images captured by the microscopes. What Globus does is provide researchers with a secure, unified interface to their research data, and manages the details of data transfers between the systems within and across institutions. For researchers using Cryo-EM data, this means they can concentrate on their research, such as investigating the secrets of how our cells respond to diseases and treatments, rather than worrying about data transfers.
AARNet and ACCS are also working with the Globus team to fine-tune high-speed large-scale characterisation data transfers into MASSIVE from international locations, including Tokyo, New Zealand and California.
Article written by AARNet in collaboration with the ACCS
