Let us first list the various challenges that many of the organizations face’s in a normal on-premises high-performance lustre based persistent files system solution or even any cross-cloud offerings and the high-level overview about some of the challenges which are being faced is listed below:
- High data backup cost
- Complex backup solution
- CAPEX based storage solution
Before discussing the solution overview just adding few notes on what a lustre file system is and in which technologies it is being used, so a Lustre file system is an open-source parallel file system that supports many requirements of leadership class HPC simulation environments. This is predominantly used for HPC cluster workload. Amazon FSx for Lustre is a fully managed service that provides cost-effective, high-performance, scalable storage for compute workloads.
Workloads that utilizes lustre storage were listed below,
- Machine learning
- High-performance computing (HPC)
- Video rendering and simulations
For this solution, we will deploy an EC2 instance to run a storage-intensive workload which is backed with AWS FSx lustre (persistent type) as a high-performance scalable storage solution. The complete details about the architecture as well as the deployment guide were listed in the upcoming sections. But apart from this storage solution can also be used for various use cases such as HPC (High-Performance Computing) workloads, Persistent storage for Containers, and Persistent storage for Containers, and Data lakes on S3.
- For a new setup, the complete networking components will be deployed in the first place as per the architecture design.
- A VPC security group needs to be created which will be specified during the FSx Lustre persistent deployment process.
- The required storage space is configured from the specific size availability of either 1.2 TiB or increments of 2.4 TiB
- We will analyze the client storage processing requirement and based on this the performance tier will be selected where the details on it are specified below:
- FSx for Lustre persistent file system allows you to choose from three deployment options, PERSISTENT-50, PERSISTENT-100, and PERSISTENT-200. Each of these deployment options comes with 50-MB/s, 100-MB/s, or 200-MB/s baseline disk throughput per TiB of file system storage, respectively.
- Configuring LZ4 compression if required
- Post this the file system can be mounted on the Linux instances which are running the storage intensive rendering systems with the help of the FSx Lustre client.
Even if a file server (FSx Lustre) becomes unavailable on a persistent file system, it is replaced automatically within minutes of failure.
Ease of accessibility
AWS FSx Lustre provides a centralized user interface on the AWS console for complete operational tasks such as and even the operations can be quickly carried out from the AWS CLI.
A wide range of storage options and features were offered from Amazon FSx for Lustre to optimize the price and performance. Like a Solid-State Drive (SSD) storage is optimized for latency-sensitive workloads or Hard Disk Drive (HDD) storage is optimized for throughput-focused workloads.
FSx is highly secured because of its ability to encrypt data at rest using a customer-managed customer master key (CMK) with the persistent file system. Even there is no CMK in place, and then also the AWS Managed CMK can be used for the encryption at rest.
Redington can help mitigate the challenges of any data storage model. Our cloud experts help design solution that is unique to your requirements. Get in touch with us to know more about our services and how we can help you.