Backing up an AWS EC2 server to Dropbox – Part Seven – Hosting WordPress on AWS Tutorial

AWS Amazon Web Services, Tutorials

Backing up an AWS EC2 server to Dropbox – Part Seven – Hosting WordPress on AWS Tutorial

Tim
January 14, 2017

Backing Up EC2 To Dropbox

Introduction

In this tutorial we’ll be setting up a backup databases from EC2 to Dropbox. You can use it to backup anything you like – logs, your entire webroot, anything really. AWS is reliable, but an offsite backup is best practice – AWS doesn’t make this easy.

We use Attic and the Dropbox-Uploader script. Attic is a open source, deduplicating, compressing backup program written in Python. The main goal of Attic is to provide an efficient and secure way to backup data. The data deduplication technique used makes Attic suitable for daily backups since only the changes are stored.

We considered using BitTorrent Sync, and detail it below, but decided against it because constantly backing up to a home PC relied on a single computer being turned on. This might be acceptable if you have another server that’s available 24/7 running BTSync.

This has been run through carefully on a new Amazon Linux instance so it should be quite accurate. If you find any problems comment below or contact us and we’ll fix them up.

 

Update June 2017

As of June 2017 I’ve started using Borg Backup. Borg is a fork of attic but it’s much more actively maintained – Attic is quite static. The installation and use is basically identical, so you can easily substitute Borg for Attic here if you so choose.

The major problem I have with this backup system is dropbox uploader doesn’t delete the files that Attic or Borg no longer need. That means I’m using more storage than is required. I don’t have a solution to this, but I’m thinking about it and might find something eventually.

It would be easy to upload to S3 or similar, but the whole point of this exercise is to have a regular, automatic backup outside AWS. I did look at BitTorrent Sync, but I didn’t want it running on the server all the time. I could probably use something like rsync, but then I’d have to run an agent on my PC and have more firewall ports open.

If anyone has any ideas please let me know in comments below.

 

Installing Attic and Dropbox Uploader

First we need to install Attic, and Python which Attic is written in (guide from here). Note that you can install from git, but that requires additional packages installed. Attic is rarely to never updated so it’s not that beneficial. You can also consider the fork “Borg Backup”.

sudo yum install python34
cd /tmp
sudo curl -O https://bootstrap.pypa.io/get-pip.py
sudo /usr/bin/python3.4 get-pip.py
cd /opt
sudo wget https://attic-backup.org/downloads/releases/0.16/Attic-0.16-linux-x86_64.tar.gz
sudo tar -xvzf Attic-0.16-linux-x86_64.tar.gz
sudo mv Attic-0.16-linux-x86_64 attic
sudo chown -R ec2-user:ec2-user attic/
cd attic

Next we install a great open source script “Dropbox Uploader” to upload the Attic repository to Dropbox

sudo yum install git
cd /opt
sudo git clone https://github.com/andreafabrizi/Dropbox-Uploader
sudo chown -R ec2-user:ec2-user Dropbox-Uploader/
cd Dropbox-Uploader
# chmod +x dropbox_uploader.sh - May not be necessary
./dropbox_uploader.sh   # Walk through and do as the script says

Create the following script, it's a handy way to update the script - it's actively maintained
vi update_from_git
git pull
Save and exit vi
chmod a+x update_from_git

Setup Database Exports

Create directories for the Attic repository and database backups

sudo mkdir -p /var/backups/attic/database1
sudo mkdir /var/backups/database_dumps
sudo chmod -R 700 /var/backups/
sudo chown -R ec2-user:ec2-user /var/backups/

Export the Databases

Setup a MySQL user that has appropriate rights to run backups

mysql -h (db-ip-or-url) -u **USER -p
(type in password)
Note that below instead of specifying an IP you can use % to indicate any IP, but this is a security risk
GRANT SELECT, LOCK TABLES ON ***INSERT_DATABASE_NAME_HERE.* TO '***backupuser'@'***PRIVATE_IP_HERE' IDENTIFIED BY '***INSERT_PASSWORD_HERE';
FLUSH PRIVILEGES;

Now we provide a way for a cron job to log into MySQL automatically, by storing the user and password in the home directory.

cd ~
vi .my.cnf

Add the following contents to the file

[mysqldump]
user=backupuser
password=abcd

Save and exit vi
chmod 600 .my.cnf

Now we’re going to create the scripts that do the database exports

sudo mkdir -p /opt/backups-scripts
sudo chown -R ec2-user:ec2-user /opt/backups-scripts
chown -R ec2-user /opt/backups-scripts
cd /opt/backup-scripts
vi export_all_databases

Add the following command to this file. You can add additional databases as required

mysqldump -h (db host IP or host name) **DBNAME > /var/backups/database/wordpress.sql

You should test this script runs and produces the expected backup file in /var/backups/database/

Setup Attic Repository/Repositories

Before you can run a backup with Attic you need to set up the repository or repositories you need.

/opt/attic/attic init /var/backups/attic/database1/database_dumps.attic

Create the Attic Backup script

Create this file and enter the text below into it

vi /opt/backups-scripts/run_attic_backup

Enter this text

REPOSITORY=/var/backups/attic/database1/database_dumps.attic
/opt/attic/attic create --stats $REPOSITORY::databases-`date +%Y-%m-%d` /var/backups/database1
# /opt/attic/attic prune -v $REPOSITORY --keep-daily=7 --keep-weekly=4 --keep-monthly=6 --keep-yearly=1

That last line is commented out. That command prunes backups so it doesn’t keep every single daily backup, but instead keeps daily backups for a week, weekly backups for a month, monthly backups for six months, then annual backups after that. I’ve commented it out because the files are deleted from the server file system but not dropbox, and it updates the Attic internal database. What I do instead is run that command using cygwin on my PC, which removes the files from dropbox, then I run it manually on the server.

Let ec2-user run this script

chmod 700 /opt/backups-scripts

Test the attic backup but running the script. If you have problems and need to tweak it you may get a message like “error archive xyz already exists”. This is just saying that the name is taken in the attic database. You can remove it with a command like this

/opt/attic/attic delete --stats /var/backups/attic/database1/database_dumps.attic::databases-2027-12-24

Automatic Backup Upload

Create the Dropbox Upload script

vi /opt/backups-scripts/dropbox_upload
Enter this text
/opt/Dropbox-Uploader/dropbox_uploader.sh -s upload /var/backups/attic /

Make the script executable
chmod 700 /opt/backups-scripts/dropbox_upload

 

Setting up Cron to Backup Databases Daily

As ec2-user, run crontab. This will run our database export script daily at 00:00, our attic backup to run at 00:15, and the dropbox upload to run at 01:00;

crontab -e

Enter the following, then save and exit

00 00 * * * /opt/backups-scripts/export_all_databases >> /var/log/cron-backups 2>&1
15 00 * * * /opt/backups-scripts/run_attic_backup >> /var/log/cron-backups 2>&1
00 01 * * * /opt/backups-scripts/dropbox_upload >> /dev/null 2>&1

 

That should be it. There may be small tweaks required, but that should get your backups exported nightly and uploaded to Dropbox.

Extending Your Backups

If you want to dump other databases it’s a simple matter of changing your export script. So long as you back them up to the same directory attic will back them up.

I also backup my Nginx logs and my webroot. Here’s the script that backs up my webroot, with a few exceptions

REPOSITORY=/var/backups/attic/webroot.attic
/opt/attic/attic create --stats $REPOSITORY::webroot-`date +%Y-%m-%d` /var/www --exclude /var/www/foldername --exclude '*.exi' --exclude 'thumbnails' \

 

I backup Nginx logs directly rather than via Attic. I only backup the gz files, which are a few days old, to prevent it trying to backup open log files.

/opt/Dropbox-Uploader/dropbox_uploader.sh -s upload /var/log/nginx/*.gz /nginx_logs/

 

Installing or Restoring Attic Backup on Windows

If you want to restore your attic backup to a Windows machine refer to this guide. Just in case it disappears here’s the core parts:

  • Install Cygwin, a Unix like shell for Windows. Make sure you install the following packages in the cygwin install: python3, python3-setuptools, gcc-g++, curl, openssh, git, openssl-devel, Cython
  • Type all the following commands. Note that if the python version changes so will the command, so just type “easy_install” then hit tab.
easy_install-3.4 pip
pip install Cython
cd /tmp
git clone https://github.com/galets/attic
cd attic
git checkout win32
python3 setup.py install
# The following line is a test
attic init /tmp/test-repository

The advantage of running under Windows is you can prune unnecessary backups, and because Windows syncs to Dropbox live it removes the unnecesary files from there as well. If this is run on Unix the files are removed from the Unix file system but not from

BitTorrent Sync

I decided not to go with BitTorrent Sync for backups, but I did get it going using this guide, and this Nginx proxy guide.

 

Facebook Comments