Category Archives: Backups

Automatic AWS Snapshots with Replication to another Region

I’ve recently started placing Ubuntu web servers up on AWS. These are pretty small systems, that don’t utilize amazon’s Database or Elastic Load Balancer features, they’re just stand alone all-in-one systems, and are relatively small.
I wanted a way to protect these systems in case amazon ever had an event where a region was down, or unstable, which occasionally does happen. If this were a larger deployment we’d have some sort of real-time database replication between availability zones, and an Elastic Load Balancer that would allow us to seamlessly fail over. In my case, I just want the comfort of knowing there is a copy of the volume in another region, and I want it to happen automatically.

I came across a few posts which had parts of what I was looking for, but not everything. I started with this awesome script by CaseyLabs which can be found here: https://github.com/CaseyLabs/aws-ec2-ebs-automatic-snapshot-bash 

I modified it slightly to provide more verbose logging, and I added a section to both the “snapshot_volumes” and “cleanup_snapshots” functions. I also modified the IAM Security Policy to allow for copying snapshots. We’ll get into all of this in a bit, but before we start FAIR WARNING: I’m not a developer, and you use this script at your own peril. It creates snapshots & copies data (which both have costs associated with them) and deletes snapshots. There are lots of things that could go wrong if you do not take the time to understand what you are doing with this script.

First things first, let’s crate the IAM Security Policy

Creating IAM Security Policy

  1. From the main AWS menu select “Identity & Access Management”.
  2. Click “Policies” in the left hand pane
  3. Click “Get Started”
  4. Click “Create Policy”
  5. Click “Select” next to “Create Your Own Policy”
  6. Enter the following:
    1. Policy Name: manage-snapshots
    2. Description: Allow Servers to create and manage snapshots of themselves
    3. Policy Document:
    4. Click “Create Policy”
  7. Click “Groups” in the left hand pane
  8. Click “Create New Group”
  9. Name the group “Snapshot_Managers”, click “Next Step”
  10. Select the group policy “manage-snapshots” and click “Next Step”
  11. Click “Create Group”
  12. Click “Users” in the left hand pane
  13. Click “Create New Users”
  14. In the “Enter User Names” box enter “snapshot-manager”
  15. Click “Create”
  16. Click “Show User Security Credentials”, note both the Access Key ID and Secret Access Key.
  17. Click “Close”
  18. Select the newly created user
  19. Click “Add User to Groups”
  20. Select “Snapshot_Managers” and then click “Add to Groups”

Install and Configure the Script

Install AWS CLI

  1. Login as your admin user
  2. Enter the following commands:

Configure AWS CLI

  1. Enter the following commands:
  2. When prompted enter the Access Key ID for the snapshot-manager account created earlier. Press Enter
  3. When prompted enter the Secret Access Key for the snapshot-manager account created earlier. Press Enter
  4. When prompted to enter the Default Region Name enter: us-west-2a (this is the availability zone that my servers are in, yours will vary)
  5. When prompted to enter Default Output Format, enter: text

Download and configure script.

The script can be downloaded and viewed from here. (rename to .sh)

Notes about the script:

  • The User’s home directory will hold the AWS CLI configuration files that directory needs to be set within the script
  • it’s hard set to wait 10 minutes between when it starts a snapshot, and when it attempts to copy that snapshot to a new region. If your snapshots are huge, this may need to be adjusted.
  • It’s configured to delete any snapshot older than the retention period, which is currently 7 days, if you want a longer retention period, this should be adjusted
  • The zone that we’re replicating the snapshots to is hard set as us-east-1, this will need adjustment if you want snapshots copied elsewhere. It also uses the description component of the remote snapshot to hold the name of the original snapshot, this is important, as when the original is deleted, that original snapshot id is used to query the remote region for snapshots whose descriptions match, and delete those as well.

Instructions

  1. Enter the following commands

Configure Cron Job

  1. Enter the following commands
  2. When prompted, select “2. Nano” as the editor. Add the following line to the end of the file:
  3. This line will run the script on minute 0, of hour 23, on every day of the month, of every month of the year, but only if that day is sunday (0), explanation below
  4. Press “Control + O” to write the file and “Control + X” to exit crontab

Sources

First and Foremost, 90% of this was written by CaseyLabs, which can be found here

Cron Job information that was the most helpful was found here

Copy-Snapshot documentation can be found here

Documentation on describing EC2 snapshots can be found here

Helpful crontab troubleshooting tips can be found here

 

When attempting to backup System State on Server 2008 R2 / SBS 2011 you get: ERROR – A Volume Shadow Copy Service operation error has occurred: (0x800423f0)

I created a script that takes a system state backup and I schedule it to run once a week so that we have a local copy of System state if we ever need it. I’m not sure that we’ve ever actually needed it, but just call me captain safety pants. About a week ago the script just stopped working, the scheduled job was failing, but the logs were pretty much empty with the exception of this one warning that was getting logged each time the script was run:

Source:        VSS
Event ID:      8230
Volume Shadow Copy Service error: Failed resolving account spsearch with status 1376. Check connection to domain controller and VssAccessControl registry key.
Operation:
Gather writers' status
Executing Asynchronous Operation
Context:
Current State: GatherWriterStatus
Error-specific details:
Error: NetLocalGroupGetMemebers(spsearch), 0x80070560, The specified local group does not exist.

I’m not sure what the spsearch user account has to do with taking a System State Backup, but it looks like that account is listed under the registry as having access to the VSS writers. The problem is it doesn’t have permissions to the writers, so when the writers start they error out saying this account does not have permissions even though they don’t need to run under this user’s context, and in fact are running under the admin user’s context (configured in the scheduled task’s job preferences).

Since this particular Server (SBS 2011) is not going to be used for sharepoint services I just removed the account from the list in the following registy entry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VSS\VssAccessControl

Once that user was removed from that registry key I restarted the scheduled task and the script completed fine. I’m not sure if this account not being in this key will cause any problems but I suspect it wont because it does not have permissions to the VSS writers anyway, which is why the script started failing in the first place.

If anyone has any spare time and wants to explain the purpose of spsearch account being in the VssAccessControl key please feel free to leave a comment.

Thanks,

Sean

Getting System State Only Backups from Server 2008 / 2008 R2

If you’re like me you’re pretty crazy about backups. I hate not having backups, it never happens that you have an error or a data loss after a successful backup, it’s always after the backup fails. That’s why I always go the extra step and backup core components with their specific tools and then backup those backups with another 3rd party backup software. In server 2000/2003 I used to use NTBackup to make 2 overlapping system state backups, one for Monday, Wednesday, Friday, and one for Tuesday and Thursday. That way if Active Directory ever crapped out on me I’d have the last two days worth of system state backups, and in the original Microsoft form to boot. (it always easiest to do the restore with Microsoft tools, ever get stuck between two vendors swearing it’s the other guy’s fault? )

Anyway, so when Server 2008 came out I wasn’t happy with the fact that I had to either backup the entire C: volume, or that I had to backup to another volume other than the once i was taking a backup of, or that the command line option wasn’t automated. Sometimes you’ve got servers in a rack that only have 1 volume, and there’s not room for a USB drive, or if you’re like me you’re backing up with a 3rd party tool and you just want to keep the native format backups on the server as a spare copy. I ended up doing a bit of research and this is how you get it done:

First you’ve got to enable the ability to backup to the same volume you’re making a backup of. In Microsoft KB 944530 it gives you instructions on how to create a registry entry that enables the ability to backup to any volume. Here is the Location and Key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\wbengine\SystemStateBackup\
Name: AllowSSBToAnyVolume
Data type: DWORD
Value data: 1

Or You can download it here: AllowSSBToAnyVolume

Once that’s been done, reboot and then you’ll be able to run a Windows Backup to any volume. Now the next time I do is create two batch scripts, one for MWF, and one for TTh, but again I’m crazy.

MWF’s Backup Batch Script:
Rmdir c:\systemstatebackups\mwf /s /q
Mkdir c:\systemstatebackups\mwf
Wbadmin start systemstatebackup –quiet –backuptarget:c:
Xcopy /s /e /c /h /y c:\windowsimagebackup\*.* c:\systemstatebackups\mwf\*.*
Rmdir c:\windowsimagebackup /s /q

and TTh’s Backup Batch Script:
Rmdir c:\systemstatebackups\tth /s /q
Mkdir c:\systemstatebackups\tth
Wbadmin start systemstatebackup –quiet –backuptarget:c:
Xcopy /s /e /c /h /y c:\windowsimagebackup\*.* c:\systemstatebackups\tth\*.*
Rmdir c:\windowsimagebackup /s /q

Schedule these to run on their proper days and there you have it, the last two days worth of system state only backups, ready to be picked up by a 3rd party backup software, or to be used to restore system state in the event of Active Directory corruption or deletion. It’s maybe a little bit unnecessary, but I’d rather have them and not need them than need them and have to deal with Symantec backup exec support trying to restore a lost CEO user account at 3 in the morning.

Attached Files:

Getting the impossible-to-use Microsoft Tools for SharePoint 2010 Farm backup to work

So it’s easy enough to get Share Point Central Administration to backup your farm, but it involves logging in every day to click the backup button, which can get tedious. I wanted a way to automate it but of course nothing was easy, after about two weeks to trying this that and the other thing, I came up with a little cheat sheet. It should be noted that I’m not a Share Point or SQL admin, and in fact I’m pretty new to working with either of them, so if you see something that could be improved on, please let me know.

The first time around in our test environment we didn’t install Share Point properly, and we tried to change the service accounts running Share Point through the Central Administration site. I’m not sure if it was bad luck, or something else, but the whole damn thing came crashing down, so please RTFM when installing new-to-you software. We  ended up going back, doing a little more planning, and performed a fresh install. I’ve decided to include some of those planning notes as pre-reqs before we go into getting the backup to work, as you’ll run into permissions problems if they’re not taken care of.

Pre Requisites Notes

Service Accounts:

svcSQL – This account is used to run all SQL related services, regardless of server.

svcSharePoint – This account is the account specified during SharePoint installation, also known as the “farm account” or “timer account” (don’t ask, I don’t understand either)

svcScheduledTask – This is the account under which the schedule task runs, when configuring the task make sure to use this account.

Installation of Applications

Okay, so when you install SQL make sure to use the svcSQL account to run all of the services.

When you install SharePoint, make sure to use the svcSharePoint account as the Farm account when installing. Also make sure to point all of your SQL needs to the SQL server and not a local database.

Assign the following permissions to the SQL databases for each service account:

User: svcSharepoint
Server Instance: SQL_SERVER\SQL_INSTANCE
Server Instance-Wide Role(s):
--dbcreator
--public
--securityadmin

Database Specific Role(s):
-SharePoint_AdminContent_{GUID}
--db_owner
--Public
--SharePoint_Shell_Access

-WSS_Content_Application_Pools
--db_owner
--Public
--SharePoint_Shell_Access

-SharePoint_Config
--db_owner
--Public
--SharePoint_Shell_Access

-WSS_Content_Application_Pools
--db_owner
--Public
--SharePoint_Shell_Access

-WSS_Content_{GUID}
--db_owner
--public

-WSS_Search_SQL_SERVER
--db_owner
--public

User: svcScheduledTask
Server Instance: SQL_SERVER\SQL_INSTANCE
Server Instance-Wide Role(s):
--Public

Database Specific Role(s):
-ALL Databases
--db_backupoperator
--public
--SharePoint_Shell_Access

Actual Backup Procedure

Now that the software is installed correctly we can take backups, if the software is not installed correctly forget about backups you’ll be lucky that central administration doesn’t implode. This is probably one of the most convoluted backup procedures you’ll ever see. Because of the combination of Batch scripts, and trying to call 1 Power Shell script from another, in combination with the fact that the Task Scheduler has a hell of a time with quotes, you end up with this, which is a scheduled task that runs 3 batch files, two of which contain Power Shell commands.

First, Create a new shared folder on the backup server, in this example we called it “data” give Full Control permissions to each service account (all three)

Then, Create a batch script called “cleanup” and put the following in it (adjust for LOCAL path)

rmdir C:\Data\SharepointBackups /s /q
mkdir c:\data\sharepointbackups

Next, Create a new Power shell script called “FarmBackup” (notepad file saved as .ps1) with the following lines (adjusted for UNC path):

Add-PSSnapin Microsoft.SharePoint.PowerShell
Backup-SPFarm -Directory \\BACKUP_SERVER\Data\sharepointbackups -BackupMethod Full

Next, Create a new batch script called “LaunchPowerShell” and put the following in it: (adjust for LOCAL path)

%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe -command "& 'C:\Data\FarmBackup.ps1' "

Next, Create a new scheduled task that runs with the following settings on each tab:
General Tab:
-Run Whether use is logged in or not
-Run with highest privileges
Triggers Tab:
-Weekly – setup with proper days and times
Actions Tab:
-Create a new action, Run a Program, and run the Cleanup.bat file
-Create a 2nd action, Run a program, and enter the following command (adjusted for LOCAL path) in the Program/Script field:
D:\data\launchpowershell.bat

Run each of the following first to verify that they work before running a test job:

Cleanup.bat
Launchpowershell.bat
FarmBackup.ps1

Failures on either mean that the job won’t run.

Finally don’t forget to backup the C:\Data\SharepointBackups folder with some backup software.