Category Archives: Linux

Automatic AWS Snapshots with Replication to another Region

I’ve recently started placing Ubuntu web servers up on AWS. These are pretty small systems, that don’t utilize amazon’s Database or Elastic Load Balancer features, they’re just stand alone all-in-one systems, and are relatively small.
I wanted a way to protect these systems in case amazon ever had an event where a region was down, or unstable, which occasionally does happen. If this were a larger deployment we’d have some sort of real-time database replication between availability zones, and an Elastic Load Balancer that would allow us to seamlessly fail over. In my case, I just want the comfort of knowing there is a copy of the volume in another region, and I want it to happen automatically.

I came across a few posts which had parts of what I was looking for, but not everything. I started with this awesome script by CaseyLabs which can be found here: 

I modified it slightly to provide more verbose logging, and I added a section to both the “snapshot_volumes” and “cleanup_snapshots” functions. I also modified the IAM Security Policy to allow for copying snapshots. We’ll get into all of this in a bit, but before we start FAIR WARNING: I’m not a developer, and you use this script at your own peril. It creates snapshots & copies data (which both have costs associated with them) and deletes snapshots. There are lots of things that could go wrong if you do not take the time to understand what you are doing with this script.

First things first, let’s crate the IAM Security Policy

Creating IAM Security Policy

  1. From the main AWS menu select “Identity & Access Management”.
  2. Click “Policies” in the left hand pane
  3. Click “Get Started”
  4. Click “Create Policy”
  5. Click “Select” next to “Create Your Own Policy”
  6. Enter the following:
    1. Policy Name: manage-snapshots
    2. Description: Allow Servers to create and manage snapshots of themselves
    3. Policy Document:
    4. Click “Create Policy”
  7. Click “Groups” in the left hand pane
  8. Click “Create New Group”
  9. Name the group “Snapshot_Managers”, click “Next Step”
  10. Select the group policy “manage-snapshots” and click “Next Step”
  11. Click “Create Group”
  12. Click “Users” in the left hand pane
  13. Click “Create New Users”
  14. In the “Enter User Names” box enter “snapshot-manager”
  15. Click “Create”
  16. Click “Show User Security Credentials”, note both the Access Key ID and Secret Access Key.
  17. Click “Close”
  18. Select the newly created user
  19. Click “Add User to Groups”
  20. Select “Snapshot_Managers” and then click “Add to Groups”

Install and Configure the Script

Install AWS CLI

  1. Login as your admin user
  2. Enter the following commands:

Configure AWS CLI

  1. Enter the following commands:
  2. When prompted enter the Access Key ID for the snapshot-manager account created earlier. Press Enter
  3. When prompted enter the Secret Access Key for the snapshot-manager account created earlier. Press Enter
  4. When prompted to enter the Default Region Name enter: us-west-2a (this is the availability zone that my servers are in, yours will vary)
  5. When prompted to enter Default Output Format, enter: text

Download and configure script.

The script can be downloaded and viewed from here. (rename to .sh)

Notes about the script:

  • The User’s home directory will hold the AWS CLI configuration files that directory needs to be set within the script
  • it’s hard set to wait 10 minutes between when it starts a snapshot, and when it attempts to copy that snapshot to a new region. If your snapshots are huge, this may need to be adjusted.
  • It’s configured to delete any snapshot older than the retention period, which is currently 7 days, if you want a longer retention period, this should be adjusted
  • The zone that we’re replicating the snapshots to is hard set as us-east-1, this will need adjustment if you want snapshots copied elsewhere. It also uses the description component of the remote snapshot to hold the name of the original snapshot, this is important, as when the original is deleted, that original snapshot id is used to query the remote region for snapshots whose descriptions match, and delete those as well.


  1. Enter the following commands

Configure Cron Job

  1. Enter the following commands
  2. When prompted, select “2. Nano” as the editor. Add the following line to the end of the file:
  3. This line will run the script on minute 0, of hour 23, on every day of the month, of every month of the year, but only if that day is sunday (0), explanation below
  4. Press “Control + O” to write the file and “Control + X” to exit crontab


First and Foremost, 90% of this was written by CaseyLabs, which can be found here

Cron Job information that was the most helpful was found here

Copy-Snapshot documentation can be found here

Documentation on describing EC2 snapshots can be found here

Helpful crontab troubleshooting tips can be found here


How to have AWS instances update Route53 CNAME each time they boot

Recently I had a need for my AWS instances to dynamically update CNAME records each time they started. You’ll only get a dedicated IP if you purchase an Elastic IP, and then, only 5 per account unless you reach out to Amazon for more. Knowing that I’m both cheap and lazy, I wanted something that would be free, as well as, automatic. I found quite a few blogs and articles that were a big help, but no one ‘put it all together’ for me. After about 6 hours I’ve got a fully working solution, but please feel free to comment on where it can be improved.

This article makes the following assumptions: Ubuntu 14.04 LTS is being used as instance OS, External DNS domain is public, and hosted on Route 53.

I’m neither an AWS nor Linux daily user, so if you see something that could be improved, please do let me know.

Create AWS User, Group, and Policy for Dynamic DNS

  1. From the main AWS menu select “Route 53”
  2. Click “Hosted Zones” in the left hand column
  3. Click “Create Hosted Zone”
  4. Enter the Domain Name that will be updated by Servers, this can be a subdomain if desired.
  5. Type: Public Hosted Zone
  6. Click “Create”
  7. Once created, note the zone ID for later.
  8. From the main AWS menu select “Identity & Access Management”.
  9. Click “Policies” in the left hand pane
  10. Click “Get Started”
  11. Click “Create Policy”
  12. Click “Select” next to “Create Your Own Policy”
  13. Enter the following:
    1. Policy Name: change-dns-records
    2. Description: Allow Servers to update their own CNAME Records each time they reboot.
    3. Policy Document:
    4. NOTE: Replace <Zone ID> with the zone ID of the DNS zone the server needs to update.
    5. Click “Create Policy”
  14. Click “Groups” in the left hand pane
  15. Click “Create New Group”
  16. Name the group “DNS_Editors”, click “Next Step”
  17. Select the group policy “change-dns-records” and click “Next Step”
  18. Click “Create Group”
  19. Click “Users” in the left hand pane
  20. Click “Create New Users”
  21. In the “Enter User Names” box enter “dns-editor”
  22. Click “Create”
  23. Click “Show User Security Credentials”, note both the Access Key ID and Secret Access Key.
  24. Click “Close”
  25. Select the newly created user
  26. Click “Add User to Groups”
  27. Select “DNS_Editors” and then click “Add to Groups”

Install and configure CLI53

  1. Grab the URL for the most recent version from here: Make sure to download the proper version (I’m using AMD64)
  2. Login to Ubuntu instance and perform the following commands (Download cli53, move to /usr/local/bin, change permissions, create a sybolic link in /usr/bin, create route53 config file, and secure it):
  3. Edit the /etc/route53/config file and enter the following:
  4. NOTE: Replace <dns-editor’s access key ID> and <dns-editor’s secret access key> with appropriate values from the dns-editor user. Update to match either your top level, or a subdomain of one of your domains.
  5. Next, create a file called /usr/sbin/ Enter the following into the file:
  6. NOTE: replace [Client_URL_ShortName] in the above text with whatever you want to CNAME to be, I use the hostname of the server, but you could use anything (www. testing. mail. )  etc.
  7. NOTE:  it should not be necessary to have to delete the record and then re-create it, the –replace flag should be able to do that in a single command, however I could not get it to work in cli53 build 6.5.0, which is what was used here. I had to delete the existing CNAME and then re-create it. I also noticed that it is case sensitive, and always created as lower case, so in your delete command you need to make sure you are specifying the record to delete in all lowercase.
  8. NOTE: in some ami distributions ec2metadata needs to be replaced with ec2-metadata
  9. Lastly we need to add the script to the logon scripts that run during boot, enter the following commands:
  10. Reboot instance and verify that it’s created a CNAME for itself.

Linux VMs running Hyper-V lose network connectivity when Live Migrated or Restarted

I ran into a problem not too long ago where a Linux VM running in Hyper-V lost network connectivity when it was restarted, or when it was Live Migrated between hosts in a Hyper-V cluster.

At first I verified all of the normal things, I made sure it was connected to the right Virtual Network, I made sure the Microsoft Network Virtual Switch Protocol was the only protocol enabled on the NIC for that Virtual Network on both Hosts in the cluster. I also made sure that the VM was using a Legacy Network Adapter instead of a Network Adapter.

I logged into the VM itself via the console and ran an ifconfig, the only adapter that showed up was lo (the loopback adapter). I thought this was odd, as I just configured eth0 not too long ago. I verified that the /etc/network/interfaces file had the proper network settings configure for eth0, which it did.

At this point I had to do some research and I found that the newer linux kernels use the MAC address of the NIC instead of the hardware ID of the device to assign a logical eth number to each network card. I checked within the VM’s Settings and sure enough the Legacy Network Adapter on the VM was set to dynamic MAC address instead of Static. I changed the setting to Static, and kept the currently assigned MAC address as it’s static MAC address.

Inside the VM I can an ifconfig -a and saw that interface eth1 was available, but not configured. I edited the /etc/network/interfaces file to say eth1 instead of eth0 and then restarted the networking services by issuing the following command: sudo /etc/init.d/networking restart. Sure enough I was able to start accessing the network.

I did a restart and a Live Migration and was able to continue accessing the network.