Monthly Archives: March 2016

Automatic AWS Snapshots with Replication to another Region

I’ve recently started placing Ubuntu web servers up on AWS. These are pretty small systems, that don’t utilize amazon’s Database or Elastic Load Balancer features, they’re just stand alone all-in-one systems, and are relatively small.
I wanted a way to protect these systems in case amazon ever had an event where a region was down, or unstable, which occasionally does happen. If this were a larger deployment we’d have some sort of real-time database replication between availability zones, and an Elastic Load Balancer that would allow us to seamlessly fail over. In my case, I just want the comfort of knowing there is a copy of the volume in another region, and I want it to happen automatically.

I came across a few posts which had parts of what I was looking for, but not everything. I started with this awesome script by CaseyLabs which can be found here: https://github.com/CaseyLabs/aws-ec2-ebs-automatic-snapshot-bash 

I modified it slightly to provide more verbose logging, and I added a section to both the “snapshot_volumes” and “cleanup_snapshots” functions. I also modified the IAM Security Policy to allow for copying snapshots. We’ll get into all of this in a bit, but before we start FAIR WARNING: I’m not a developer, and you use this script at your own peril. It creates snapshots & copies data (which both have costs associated with them) and deletes snapshots. There are lots of things that could go wrong if you do not take the time to understand what you are doing with this script.

First things first, let’s crate the IAM Security Policy

Creating IAM Security Policy

  1. From the main AWS menu select “Identity & Access Management”.
  2. Click “Policies” in the left hand pane
  3. Click “Get Started”
  4. Click “Create Policy”
  5. Click “Select” next to “Create Your Own Policy”
  6. Enter the following:
    1. Policy Name: manage-snapshots
    2. Description: Allow Servers to create and manage snapshots of themselves
    3. Policy Document:
    4. Click “Create Policy”
  7. Click “Groups” in the left hand pane
  8. Click “Create New Group”
  9. Name the group “Snapshot_Managers”, click “Next Step”
  10. Select the group policy “manage-snapshots” and click “Next Step”
  11. Click “Create Group”
  12. Click “Users” in the left hand pane
  13. Click “Create New Users”
  14. In the “Enter User Names” box enter “snapshot-manager”
  15. Click “Create”
  16. Click “Show User Security Credentials”, note both the Access Key ID and Secret Access Key.
  17. Click “Close”
  18. Select the newly created user
  19. Click “Add User to Groups”
  20. Select “Snapshot_Managers” and then click “Add to Groups”

Install and Configure the Script

Install AWS CLI

  1. Login as your admin user
  2. Enter the following commands:

Configure AWS CLI

  1. Enter the following commands:
  2. When prompted enter the Access Key ID for the snapshot-manager account created earlier. Press Enter
  3. When prompted enter the Secret Access Key for the snapshot-manager account created earlier. Press Enter
  4. When prompted to enter the Default Region Name enter: us-west-2a (this is the availability zone that my servers are in, yours will vary)
  5. When prompted to enter Default Output Format, enter: text

Download and configure script.

The script can be downloaded and viewed from here. (rename to .sh)

Notes about the script:

  • The User’s home directory will hold the AWS CLI configuration files that directory needs to be set within the script
  • it’s hard set to wait 10 minutes between when it starts a snapshot, and when it attempts to copy that snapshot to a new region. If your snapshots are huge, this may need to be adjusted.
  • It’s configured to delete any snapshot older than the retention period, which is currently 7 days, if you want a longer retention period, this should be adjusted
  • The zone that we’re replicating the snapshots to is hard set as us-east-1, this will need adjustment if you want snapshots copied elsewhere. It also uses the description component of the remote snapshot to hold the name of the original snapshot, this is important, as when the original is deleted, that original snapshot id is used to query the remote region for snapshots whose descriptions match, and delete those as well.

Instructions

  1. Enter the following commands

Configure Cron Job

  1. Enter the following commands
  2. When prompted, select “2. Nano” as the editor. Add the following line to the end of the file:
  3. This line will run the script on minute 0, of hour 23, on every day of the month, of every month of the year, but only if that day is sunday (0), explanation below
  4. Press “Control + O” to write the file and “Control + X” to exit crontab

Sources

First and Foremost, 90% of this was written by CaseyLabs, which can be found here

Cron Job information that was the most helpful was found here

Copy-Snapshot documentation can be found here

Documentation on describing EC2 snapshots can be found here

Helpful crontab troubleshooting tips can be found here