AWS storage services

Introduction

Amazon offers several storage services, each optimized for specific use cases. In order to choose the best storage for your application, first we must understand the various offerings.

In this article we will briefly discuss Amazon Simple Storage Service (S3), Elastic Block Store (EBS) and Elastic File Store (EFS). We will cover optimal use cases for each service and show how to integrate Amazon storage with your on-premise solutions.

S3 basics and terminology

Amazon S3 is a highly durable and highly available object storage service. As object-based storage, there is no need to provision a specific amount of space to store your data on, like a standard hard drive (block storage). You may upload any amount of data to S3 without worrying about running out of storage, and you will only be charged for the data you are currently using.

In order to upload data to S3, you must first create “buckets” in a specific AWS region. A bucket is a container for your data or objects. Access to each bucket can be controlled using an S3 bucket policy or IAM policy. By default, S3 buckets are private (only the AWS account that created the bucket can access the bucket). 

One thing to note: bucket names are globally unique across all AWS accounts. This means that only one bucket named “Files” or “Backups” may exist across ALL AWS accounts. You will not be able to create a bucket with the same name as any other bucket used by any other account.

Data stored in S3 is known as an object, a key-value pair. A key is the name of the file, while the value is the specific data you are storing. Each object stored in an S3 bucket may be up to 5 TB in size.

The final pieces of information commonly discussed when discussing S3 are durability and availability. Durability is the chance of losing data to drive failures or corruption. All storage classes of S3 are designed for 99.999999999% durability (also known as 11 nines durability, only a 0.000000001% chance of losing data). Availability is the chance to retrieve the data at the exact moment needed. All S3 classes are designed for between 99.5 and 99.99% availability (see below).

There are different classes of storage within S3, each with their own use:

• S3 Standard: S3 standard data is available instantly and is commonly used for content distribution, dynamic websites and big data analytics. 99.99% availability
• S3 Standard Infrequent Access (IA) and One Zone Infrequent Access: S3 IA and One Zone IA have the low latency and high throughput of Standard but with a lower cost per GB of storage and a per GB retrieval fee. S3 IA is best used for backups and disaster recovery, or other data which is not required often but when it is, is needed quickly. One Zone IA is entirely hosted in a single availability zone and should be used for data that can be easily recreated, such as certain reports. 99.9% availability for IA, 99.5% availability for One Zone IA
• S3 Glacier and Glacier Deep Archive: Glacier and Deep Archive is a low-cost archival space for objects, where instant availability is not needed. Retrieval time ranges from minutes to two days. 99.99% availability
• S3 Intelligent Tiering: S3 intelligent tiering will optimize costs by automatically transitioning objects between S3 Standard and S3 Infrequent Access. 99.9% availability

S3 also offers transfer acceleration for uploading files. When transfer acceleration is enabled, the file upload will enter Amazon’s network at the nearest endpoint to the user and travel to your S3 bucket entirely through Amazon’s network. This can speed up your uploads by 50 to 500% for larger uploads. 

Elastic Block Store (EBS) and Elastic File System (EFS)

In addition to S3’s object-based storage, AWS also offers two block storage services, Elastic Block Store and Elastic File System. Both EBS and EFS volumes can be mounted to your EC2 instances as a standard volume but they function very differently.

Elastic Block Store (EBS) allows you to create specific volumes to attach to EC2 instances. Each volume may only be attached to a single EC2 instance. When creating an EBS volume, you specify an amount of storage (say, 100 GB) just like provisioning a local drive for an on-premise server. If you fill the entire volume, you will either need to remove data or expand the volume to continue using it.

Elastic File Store (EFS) volumes, on the other hand, are network file share volumes. They may be mounted to multiple EC2 instances at the same time. This allows multiple instances to write and pull information from the same volume simultaneously. Unlike EBS (where you specify an amount of storage when you provision the volume), EFS is expandable. You are only charged for the storage you use. 

One limiting factor for EFS is that these volumes cannot be mounted to Windows instances. 

Integrating on-premises storage

AWS allows a hybrid cloud storage option with AWS Storage Gateway. In order to use Storage Gateway, you must deploy a virtual machine or gateway hardware appliance within your on-premise environment. Once you have deployed the Storage Gateway, you will be able to integrate with other AWS storage services such as S3, Glacier and EBS.

Three different types of gateways are available: File, Tape and Volume Gateway. File gateways are used for backing up an on-premises file share in the cloud or creating a local cache for cloud storage. Tape gateways provide a virtual tape library and easily integrates with other backup applications. Volume gateways allow for snapshots of local volumes and drives to be stored as an EBS volume to be used as a backup or attached to Amazon EC2 instances.

Conclusion

Between S3, EBS and EFS, AWS offers many options for storage, both object and block-level. Depending on your needs, you will find a storage option to your workload, either fully in the cloud or hybrid with your on-premises system.

 

Sources

1. What is Amazon S3?, AWS
2. Amazon S3 Storage Classes, AWS
3. Amazon Elastic Block Store (Amazon EBS), AWS
4. Amazon Elastic File System (Amazon EFS), AWS
5. AWS Storage Gateway, AWS