Challenges
Let’s start with a high-level overview of some of the problems that many firms have in a typical on-premises high-performance Lustre-based persistent files system solution or even any Hybrid Cloud services:
- High data backup cost
- Complex backup solution
- CAPEX based storage solution
Solution overview
Before we go into the solution overview, let’s define what a Lustre file system is and how it’s used in different technologies. A Lustre file system is an open-source parallel file system that covers many of the criteria of leadership class HPC simulation environments. This is mostly utilized for workloads on HPC clusters. Amazon FSx for Lustre is a fully managed storage service for compute workloads that is cost-effective, high-performance, and scalable.
Workloads that utilize Lustre storage were listed below:
- Machine learning
- High-performance computing (HPC)
- Video rendering and simulations
We’ll use an Amazon Elastic Compute Cloud (Amazon EC2) instance to run a storage-intensive job, which will be backed by Amazon FSx for Lustre (persistent type) as a high-performance, scalable storage solution. In the next parts, we’ll go through the architecture in detail as well as the deployment method. Aside from that, this storage solution may be used for a variety of use cases, including HPC workloads, Persistent storage for Containers, and Data lakes on Amazon Simple Storage Service (Amazon S3).
Architecture
Deployment
- For a new setup, the complete networking components will be deployed in the first place as per the above architecture design
- Amazon Virtual Private Cloud (Amazon VPC) security group needs to be created which will be specified during the Amazon FSx for Lustre persistent deployment process
- The required storage space is configured from the specific size availability of either 1.2 TiB or increments of 2.4 TiB
- We will analyze the client storage processing requirement and based on this the performance tier will be selected where the details on it are specified below:
- Amazon FSx for Lustre persistent file system allows you to choose from three deployment options, PERSISTENT-50, PERSISTENT-100, and PERSISTENT-200. Each of these deployment options comes with 50-MB/s, 100-MB/s, or 200- MB/s baseline disk throughput per TiB of file system storage, respectively
- Configuring LZ4 compression if required
- Post this the file system can be mounted on the Linux instances which are running the storage-intensive rendering systems with the help of Amazon FSx for Lustre client
Benefits
Better redundancy
Even if a file server (Amazon FSx for Lustre) becomes unavailable on a persistent file system, it is replaced automatically within minutes of failure.
Ease of accessibility
Amazon FSx for Lustre provides a centralized user interface on the AWS console for complete operational tasks and even the operations can be quickly carried out from the AWS Command Line Interface (AWS CLI).
Cost Optimization
A wide range of storage options and features were offered from Amazon FSx for Lustre to optimize the price and performance. Storage on Solid-State Drives (SSDs) is optimized for latency-sensitive applications, whereas storage on Hard Disk Drives (HDDs) is intended for throughput-focused workloads.
Security
Amazon FSx for Lustre is highly secured because of its ability to encrypt data at rest using customer managed Customer Master Key (CMK) with the persistent file system. Even if there is no CMK in place, AWS Managed CMK can be used for encryption at rest.
Redington can help mitigate the challenges of any data storage model. Our cloud experts help design solution that is unique to your requirements. Get in touch with us to know more about our services and how we can help you.
