Backing Up EC2 To Dropbox
Introduction
In this tutorial we’ll be setting up a backup databases from EC2 to Dropbox. You can use it to backup anything you like – logs, your entire webroot, anything really. AWS is reliable, but an offsite backup is best practice – AWS doesn’t make this easy.
We use Attic and the Dropbox-Uploader script. Attic is a open source, deduplicating, compressing backup program written in Python. The main goal of Attic is to provide an efficient and secure way to backup data. The data deduplication technique used makes Attic suitable for daily backups since only the changes are stored.
We considered using BitTorrent Sync, and detail it below, but decided against it because constantly backing up to a home PC relied on a single computer being turned on. This might be acceptable if you have another server that’s available 24/7 running BTSync.
This has been run through carefully on a new Amazon Linux instance so it should be quite accurate. If you find any problems comment below or contact us and we’ll fix them up.
Update June 2017
As of June 2017 I’ve started using Borg Backup. Borg is a fork of attic but it’s much more actively maintained – Attic is quite static. The installation and use is basically identical, so you can easily substitute Borg for Attic here if you so choose.
The major problem I have with this backup system is dropbox uploader doesn’t delete the files that Attic or Borg no longer need. That means I’m using more storage than is required. I don’t have a solution to this, but I’m thinking about it and might find something eventually.
It would be easy to upload to S3 or similar, but the whole point of this exercise is to have a regular, automatic backup outside AWS. I did look at BitTorrent Sync, but I didn’t want it running on the server all the time. I could probably use something like rsync, but then I’d have to run an agent on my PC and have more firewall ports open.
If anyone has any ideas please let me know in comments below.
Installing Attic and Dropbox Uploader
First we need to install Attic, and Python which Attic is written in (guide from here). Note that you can install from git, but that requires additional packages installed. Attic is rarely to never updated so it’s not that beneficial. You can also consider the fork “Borg Backup”.
sudo yum install python34 cd /tmp sudo curl -O https://bootstrap.pypa.io/get-pip.py sudo /usr/bin/python3.4 get-pip.py cd /opt sudo wget https://attic-backup.org/downloads/releases/0.16/Attic-0.16-linux-x86_64.tar.gz sudo tar -xvzf Attic-0.16-linux-x86_64.tar.gz sudo mv Attic-0.16-linux-x86_64 attic sudo chown -R ec2-user:ec2-user attic/ cd attic
Next we install a great open source script “Dropbox Uploader” to upload the Attic repository to Dropbox
sudo yum install git cd /opt sudo git clone https://github.com/andreafabrizi/Dropbox-Uploader sudo chown -R ec2-user:ec2-user Dropbox-Uploader/ cd Dropbox-Uploader # chmod +x dropbox_uploader.sh - May not be necessary ./dropbox_uploader.sh # Walk through and do as the script says Create the following script, it's a handy way to update the script - it's actively maintained vi update_from_git git pull Save and exit vi chmod a+x update_from_git
Setup Database Exports
Create directories for the Attic repository and database backups
sudo mkdir -p /var/backups/attic/database1 sudo mkdir /var/backups/database_dumps sudo chmod -R 700 /var/backups/ sudo chown -R ec2-user:ec2-user /var/backups/
Export the Databases
Setup a MySQL user that has appropriate rights to run backups
mysql -h (db-ip-or-url) -u **USER -p (type in password) Note that below instead of specifying an IP you can use % to indicate any IP, but this is a security risk GRANT SELECT, LOCK TABLES ON ***INSERT_DATABASE_NAME_HERE.* TO '***backupuser'@'***PRIVATE_IP_HERE' IDENTIFIED BY '***INSERT_PASSWORD_HERE'; FLUSH PRIVILEGES;
Now we provide a way for a cron job to log into MySQL automatically, by storing the user and password in the home directory.
cd ~ vi .my.cnf
Add the following contents to the file
[mysqldump] user=backupuser password=abcd Save and exit vi
chmod 600 .my.cnf
Now we’re going to create the scripts that do the database exports
sudo mkdir -p /opt/backups-scripts sudo chown -R ec2-user:ec2-user /opt/backups-scripts chown -R ec2-user /opt/backups-scripts cd /opt/backup-scripts vi export_all_databases
Add the following command to this file. You can add additional databases as required
mysqldump -h (db host IP or host name) **DBNAME > /var/backups/database/wordpress.sql
You should test this script runs and produces the expected backup file in /var/backups/database/
Setup Attic Repository/Repositories
Before you can run a backup with Attic you need to set up the repository or repositories you need.
/opt/attic/attic init /var/backups/attic/database1/database_dumps.attic
Create the Attic Backup script
Create this file and enter the text below into it
vi /opt/backups-scripts/run_attic_backup Enter this text REPOSITORY=/var/backups/attic/database1/database_dumps.attic /opt/attic/attic create --stats $REPOSITORY::databases-`date +%Y-%m-%d` /var/backups/database1 # /opt/attic/attic prune -v $REPOSITORY --keep-daily=7 --keep-weekly=4 --keep-monthly=6 --keep-yearly=1
That last line is commented out. That command prunes backups so it doesn’t keep every single daily backup, but instead keeps daily backups for a week, weekly backups for a month, monthly backups for six months, then annual backups after that. I’ve commented it out because the files are deleted from the server file system but not dropbox, and it updates the Attic internal database. What I do instead is run that command using cygwin on my PC, which removes the files from dropbox, then I run it manually on the server.
Let ec2-user run this script
chmod 700 /opt/backups-scripts
Test the attic backup but running the script. If you have problems and need to tweak it you may get a message like “error archive xyz already exists”. This is just saying that the name is taken in the attic database. You can remove it with a command like this
/opt/attic/attic delete --stats /var/backups/attic/database1/database_dumps.attic::databases-2027-12-24
Automatic Backup Upload
Create the Dropbox Upload script
vi /opt/backups-scripts/dropbox_upload Enter this text /opt/Dropbox-Uploader/dropbox_uploader.sh -s upload /var/backups/attic / Make the script executable chmod 700 /opt/backups-scripts/dropbox_upload
Setting up Cron to Backup Databases Daily
As ec2-user, run crontab. This will run our database export script daily at 00:00, our attic backup to run at 00:15, and the dropbox upload to run at 01:00;
crontab -e
Enter the following, then save and exit
00 00 * * * /opt/backups-scripts/export_all_databases >> /var/log/cron-backups 2>&1 15 00 * * * /opt/backups-scripts/run_attic_backup >> /var/log/cron-backups 2>&1 00 01 * * * /opt/backups-scripts/dropbox_upload >> /dev/null 2>&1
That should be it. There may be small tweaks required, but that should get your backups exported nightly and uploaded to Dropbox.
Extending Your Backups
If you want to dump other databases it’s a simple matter of changing your export script. So long as you back them up to the same directory attic will back them up.
I also backup my Nginx logs and my webroot. Here’s the script that backs up my webroot, with a few exceptions
REPOSITORY=/var/backups/attic/webroot.attic /opt/attic/attic create --stats $REPOSITORY::webroot-`date +%Y-%m-%d` /var/www --exclude /var/www/foldername --exclude '*.exi' --exclude 'thumbnails' \
I backup Nginx logs directly rather than via Attic. I only backup the gz files, which are a few days old, to prevent it trying to backup open log files.
/opt/Dropbox-Uploader/dropbox_uploader.sh -s upload /var/log/nginx/*.gz /nginx_logs/
Installing or Restoring Attic Backup on Windows
If you want to restore your attic backup to a Windows machine refer to this guide. Just in case it disappears here’s the core parts:
- Install Cygwin, a Unix like shell for Windows. Make sure you install the following packages in the cygwin install: python3, python3-setuptools, gcc-g++, curl, openssh, git, openssl-devel, Cython
- Type all the following commands. Note that if the python version changes so will the command, so just type “easy_install” then hit tab.
easy_install-3.4 pip pip install Cython cd /tmp git clone https://github.com/galets/attic cd attic git checkout win32 python3 setup.py install # The following line is a test attic init /tmp/test-repository
The advantage of running under Windows is you can prune unnecessary backups, and because Windows syncs to Dropbox live it removes the unnecesary files from there as well. If this is run on Unix the files are removed from the Unix file system but not from
BitTorrent Sync
I decided not to go with BitTorrent Sync for backups, but I did get it going using this guide, and this Nginx proxy guide.