Bridging HPC and Cloud storage
AKQUINET, MT-C and the Fenix consortium have teamed-up to enhance the multifunctional Data Mover NODEUM® for realising a data mover service within the Fenix infrastructure. This service will allow users to seamlessly manage data stored in high-performance parallel file systems as well as on-premise Cloud data repositories.
The Jülich Supercomputing Centre (JSC) as representative of the Fenix consortium has decided to commission AKQUINET and MT-C with development tasks for the implementation of part of the Fenix ??infrastructure. The aim is to simplify the data transfer between cloud object stores with a Swift interface and fast parallel file systems with a POSIX interface. For this purpose, the multifunctional data mover NODEUM® is to be used with the corresponding expansion of specific functionalities.
The goal of the project is to realise the Data Mover service at different Fenix ??locations, including the European supercomputing centres JSC (Germany), BSC (Spain), CINECA (Italy), CEA (France) and CSCS (Switzerland). Federated cloud object stores are used at these locations, which enable researchers to exchange their data. With the Data Mover, researchers can copy their data locally to the existing parallel file systems in order to process them on the fast HPC systems. In addition, researchers can copy data, which have been generated on these supercomputers to the cloud object stores in order to make them accessible to other researchers from all over the world.
Today the main storage service provided by HPC centres is realised by internal but super-fast parallel file systems based on, e.g., GPFS or Lustre. They thus work as individual silos in internal infrastructures. To make data externally accessible and sharable, cloud object stores technologies are better suitable. The scope of the project is to connect the two worlds with the NODEUM® Data Mover and to transfer data from the world of the parallel file systems to the new world of Posix / S3 – considering all AAI requirements.
In addition to the DataMover, AKQUINET is increasingly establishing itself as a supplier of hardware in the HPC environment. In addition to the storage solutions from Quantum and Cloudian, AMD EPYC and NVidia-based server systems for AI / KI are increasingly being used.
