With Amazon Web Services’ (AWS) Storage Gateway-stored volumes and VMWare ESXi, you can automatically and asynchronously sync data to an Amazon AWS S3 volume over a Secure Socket Layer (SSL) channel. The gateway is provided in the form of an AWS .ova virtual machine template that runs on VMware ESXi.

The AWS virtual machine shares two iSCSI Volumes with your on-premises Windows server and the cloud. If you’re proficient with VMWare ESXi, you can easily sync local files with Amazon AWS and sync your on-premises data with the cloud.

Getting started ^

This article will assume that you already have an AWS account and you have VMWare ESXi running with a Windows VM also running on your host. Your Storage Gateway VM will need 7.5 GB of memory, and its two additional disks will require their own datastore. Storage Gateways aren’t exactly cheap at $125/month, but the first 60 days after activation are free. While it may sound like a lot of money, a service such as Iron Mountain offsite storage will cost a similar amount but with less flexibility.

First, we’re going to log into AWS, and then start the Storage Gateway setup. While there are several storage options available (for example, a virtual tape disk), we’ll stick to setting up synched volumes, which are called “Gateway-stored volumes.” Do this by navigating to the Storage & Content Delivery section of AWS, and select Storage Gateway:

Select Storage Gateway

Select Storage Gateway

The following page will describe the Storage Gateway and the general procedure for setting up the gateway. A wizard will ask you to Setup and Activate a New Gateway.

Set up and activate a new gateway

Set up and activate a new gateway

We’re now presented with three options: Gateway-Cached volumes, Gateway-Virtual Tape Library, and Gateway-Stored Volumes: Schedule off-site backups to Amazon S3 for your on-premises data. Select the third option, and click Continue.

Select gateway-stored volumes

Select gateway-stored volumes

After a warning, we’re presented with three different virtual machines to download. Select the first option, “I want to run the AWS Storage Gateway on VMware ESXi,” and then click Continue.

Select VMware ESXi

Select VMware ESXi

On the following page, click Download to download the AWS Storage Gateway VM software. The download is a zip file approximately 900 MB in size.

Download the VM

Download the VM

After unzipping the .ova file, be sure to place the file on a computer where the VSphere Client is installed. Log into your VSphere Client with administrative privileges. Similarly, click Continue in the Storage Gateway setup wizard.

Building the virtual machine ^

Log into vSphere

Log into vSphere

In vSphere, click on the File drop-down menu, and select Deploy OVF Template. Point the installer to your downloaded and unzipped .ova file. Name the storage gateway.

Name the storage gateway

Name the storage gateway

Select a datastore where you want to store the .ova.

Specify datastore

Specify datastore

When prompted for Disk Format, choose Thick provisioned format, and then click Finish.

Complete OVF deployment

Complete OVF deployment

It could take several minutes for the deployment to complete. Once you have completed the deployment, the new VM will appear on your ESXi host.

It’s important that your ESXi host has its clock synchronized with a Network Time Protocol (NTP) server. Additionally, we want to be certain that the new virtual machine uses the host to synchronize its own clock. Right-click on the Storage Gateway VM, and click Edit Settings. In the Options tab, select the VMware Tools setting in the Settings column. In the Advanced section, select Synchronize guest time with host, and click OK.

Sync guest time with host

Sync guest time with host

Provision local disks ^

Here we’ll provision local storage with two new volumes. For more detailed step-by-step instructions, refer to the documentation here. In short, we’ll create two disks inside the Storage Gateway VM settings. Once again, use Thick Provisioning, and ensure that the disks use Paravirtualized Disk Controllers. Keep in mind that you’ll first create the two disks and then enable the Paravirtualized disk controllers. The disks can be anywhere from 1 GB to 1 TB in size. It is recommended that you use a datastore separate from the one in which the virtual machine is located, as demonstrated in the following screenshot.

Specify datastore

Specify datastore

Once the disks have been created, right-click on the Storage Gateway VM, and click Edit Settings. Select the SCSI controller, and click Change Type.

Change SCSI controller type

Change SCSI controller type

Select VMware Paravirtual

Select VMware Paravirtual

Activate the Storage Gateway ^

Back in the AWS Setup and Activate Gateway wizard, after configuring your disks, click Continue to continue with activating the gateway.

AWS allocate local disks

AWS allocate local disks

Power up your Storage Gateway VM, go to the Summary tab, and copy the IP address.

Copy new IP address

Copy new IP address

Paste the VM’s IP address into the AWS wizard. It’s OK that the IP address is one that came from your local network—you do not need to enter your external IP here or mess around with a firewall. Then, click Proceed to Activation.

Enter IP address and proceed

Enter IP address and proceed

You’ll now be prompted to enter your time zone, provide a Gateway Name, and then Activate My Storage Gateway.

Activate storage gateway

Activate storage gateway

Now that the Storage Gateway has been activated, we want to Create Volumes in the AWS console to begin using the AWS Storage Gateway. Note that if we want to shut down or delete our Storage Gateway, we’ll do that under the Gateway tab in this screen.

Storage gateway settings

Storage gateway settings

Create AWS storage volumes ^

Under the Volumes tab, click Create Volumes, select one of your SCSI disks, name the volume, and click Create Volume.

Create first volume

Create first volume

Next, we will configure the iSCSI settings. Find your unique initiator name in your Windows Server iSCSI Initiator Configuration tab and enter it, enter an initiator secret and a Mutual CHAP secret, and click Save. If you’re unsure how to configure CHAP for iSCSI, consult the documentation here. In the interest of experimentation, you can skip CHAP authentication altogether.

Enter IP address and proceed

Enter IP address and proceed

While creating your volumes, select which volume will be used as the upload buffer and which will be the cache volume.

Configure local upload buffer

Configure local upload buffer

When we’ve completed the creation of our storage volumes, AWS will notify us that our volume is available.

Volume available

Volume available

Configure Windows iSCSI Initiator ^

Back on your Windows server, configure your CHAP secret.

Enter IP address and proceed

Enter IP address and proceed

When your volumes and CHAP are ready, connect to your volumes from the Discovery tab, then click Discover Portal, enter your IP address of your Storage Gateway, and then click the Advanced button.

iSCSI Discover Portal

iSCSI Discover Portal

Place a checkmark in Enable CHAP login, specify the Name, Target Secret, and place a checkmark in Perform mutual authentication.

Connect to the Storage Gateway volume ^

Next we’ll go into the Targets tab and connect to the IP address of our local Storage Gateway. If all goes well, you should see you’re now connected.

iSCSI discovered target

iSCSI discovered target

We are now able to go into the iSCSI Initiator Volumes and Devices tab and click the AutoConfigure button, which will populate the volume in the Volume List.

Auto configure iSCSI volumes

Auto configure iSCSI volumes

Initialize the iSCSI volume in Windows ^

We can finally head into Windows Computer Management tool, click on Disk Management, initialize our new disk, provide a drive letter, and begin synching data with our AWS volume.

Subscribe to 4sysops newsletter!

Initialize AWS volume

Initialize AWS volume

Where to go from here? ^

Now that we’ve got our Storage Gateway configured, we can manage our activated gateway, set bandwidth rate limits, optimize performance, take snapshots, and use those snapshots to build new EC2 instances. If your organization is already using AWS, it may prove worthwhile to bridge the storage gap between your on-premises datacenter and AWS S3 with Storage Gateways.

+1
2 Comments
  1. Ankitkumar Patel 4 years ago

    How do you integrate with S3 buckets?

    +1

    • Author

      There are 2 types of gateways, Volume and File. The differences between these two can be found in the Q&A FAQ found here and perhaps this may answer your question:

      Q: When I look in Amazon S3 why can’t I see my volume data?

      Your volumes are stored in Amazon S3 and accessible through AWS Storage Gateway. You cannot directly access them by using Amazon S3 API actions. You can take point-in-time snapshots of gateway volumes that are made available in the form of Amazon EBS snapshots. Use the file interface to work with your data natively in S3.

      +2

Leave a reply

Please enclose code in pre tags

Your email address will not be published. Required fields are marked *

*

© 4sysops 2006 - 2021

CONTACT US

Please ask IT administration questions in the forums. Any other messages are welcome.

Sending

Log in with your credentials

or    

Forgot your details?

Create Account