using duplicity

 2011-07-20

A while ago, I posted about how I backup my server with Duplicity to Amazon’s S3 storage.

To follow up, here is a little guide I wrote on using Duplicity in the everyday work environment

Overview

Duplicity is a backup tool that will create compressed and encrypted (uses gnupg) backup archives. It can use a variety of protocols as the target (file, ftp, webdav, imap, ssh/scp, rsync, hsi, s3 and hsi).

Since it is aware of previous backup jobs, incremental backups from that last full, and some basic collection management, it is preferred over simpler tools like rsync.

Since Duplicity uses GnuPG to compress and encrypt the save sets, you’ll need to enter a passphrase. This make duplicity a little difficult to automate, you can do one of two things:

embed the the gnupg passphrase in the backup job, or use a gnupg-agent
disable encryption and just compress the save set

To only compress and not encrypt, use the ‘’’–no-encryption’’’ option.

Installation

Duplicity is available in the EPEL channel, so on RHEL 5 systems just type

yum install duplicity

On Ubuntu, you can install it using aptitude:

aptitude install duplicity

On FreeBSD, you can install it using:

pkg_add -r duplicity

Preparation

You’ll need a location to store your archives. Lets assume it is a locally mounted volume at /backups.

Usage

The syntax is pretty simple:

duplicity [full|incr|collection-status|list-current-files] source target-url

The ’target-url’ can be one of many protocols:

file:///backups
scp://user@host/backups
webdav://user@backup-server/backups/

Since we are assuming a locally mounted device, we’ll use the ‘‘‘file:///’’’ url.

Full Backups

duplicity full ~/ file:///backups/

GnuPG passphrase:
Local and Remote metadata are synchronized, no sync needed.

date = "" Retype passphrase to confirm:

--------------[ Backup Statistics ]--------------
StartTime 1274121911.36 (Mon May 17 11:45:11 2010)
EndTime 1274126391.50 (Mon May 17 12:59:51 2010)
ElapsedTime 4480.14 (1 hour 14 minutes 40.14 seconds)
SourceFiles 53369
SourceFileSize 43240385480 (40.3 GB)
NewFiles 53369
NewFileSize 43240385480 (40.3 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 53369
RawDeltaSize 43219504708 (40.3 GB)
TotalDestinationSizeChange 37539659285 (35.0 GB)
Errors 0
-------------------------------------------------

Incremental Backup

Incremental backups are easy, just replace ‘full’ with ‘incr’, and here, we also upped the Gzip compression level to ‘9’

duplicity incr --gpg-options "-z 9" /home/mcarlson file:///media/0654-E203/backups
GnuPG passphrase:
Local and Remote metadata are synchronized, no sync needed.

date = "" ————–[ Backup Statistics ]————– StartTime 1274130893.40 (Mon May 17 14:14:53 2010) EndTime 1274131032.77 (Mon May 17 14:17:12 2010) ElapsedTime 139.36 (2 minutes 19.36 seconds) SourceFiles 53437 SourceFileSize 43628122482 (40.6 GB) NewFiles 172 NewFileSize 434867245 (415 MB) DeletedFiles 96 ChangedFiles 58 ChangedFileSize 1175654577 (1.09 GB) ChangedDeltaSize 0 (0 bytes) DeltaEntries 326 RawDeltaSize 444524537 (424 MB) TotalDestinationSizeChange 442395156 (422 MB) Errors 0 ————————————————-

Restore

You can either restore the entire archive with the ‘‘‘restore’’’ option, or a set of files with ‘‘‘files-to-restore’’’:

Lets assume we did something like this:

rm -rf FreeBSD

This was obviously a mistake and we quickly need those files back:

duplicity --file-to-restore src/FreeBSD file:///backups/ FreeBSD
GnuPG passphrase:
Local and Remote metadata are synchronized, no sync needed.

date = ""

Lets verify that our files are in fact there again:

ls -al FreeBSD
total 24
drwxr-xr-x  6 mcarlson mcarlson 4096 2009-08-31 15:51 .
drwxrwxr-x 65 mcarlson mcarlson 4096 2010-05-17 16:23 ..
drwxr-xr-x  2 mcarlson mcarlson 4096 2009-08-31 15:51 6.4
drwxr-xr-x  2 mcarlson mcarlson 4096 2009-08-31 15:25 7.0
drwxr-xr-x  2 mcarlson mcarlson 4096 2009-08-31 15:50 8.0
drwxr-xr-x  6 mcarlson mcarlson 4096 2009-08-31 15:52 i386

Managing your Archive(s)

Duplicity does not store any policy or retention details, so it is up the the individual to remove older save sets.

Collection Status

First thing you can do is run get a status of your save sets. Duplicity will scan your target-url, in this case ‘file:///backups’ and print a simple report on what backups types have ran, the date it was ran at, and the number of files:

duplicity collection-status file:///media/0654-E203/backups

Local and Remote metadata are synchronized, no sync needed.

date = "" Collection Status —————– Connecting with backend: LocalBackend Archive dir: /home/mcarlson/.cache/duplicity/0e6da04424dcfd02a2dae71879be23fa

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Mon May 17 11:45:04 2010
Chain end time: Mon May 17 14:14:48 2010
Number of contained backup sets: 2
Total number of contained volumes: 1447
 Type of backup set:                            Time:      Num volumes:
                Full         Mon May 17 11:45:04 2010              1430
         Incremental         Mon May 17 14:14:48 2010                17
-------------------------
No orphaned or incomplete backup sets found.

Collection Pruning

You can remove older backup archives with ‘‘‘remove-older-than’’’ or ‘‘‘remove-all-but-n-full’’’, and you can clean out failed save sets with the ‘‘‘cleanup’’’ options.

All of these require the target-url as the 2nd argument.

Here, we will remove any archive that is older than 1 month (1M).

duplicity remove-older-than 1M file:///backups/

date = "" There are backup set(s) at time(s): Thu Apr 1 00:15:17 2010 Sun Apr 4 00:01:23 2010 Sun Apr 11 00:01:29 2010 Which can’t be deleted because newer sets depend on them. Deleting backup sets at times: Sun Apr 26 12:15:33 2009 Sun Apr 26 14:45:25 2009 Sun Apr 26 14:58:38 2009 Sun May 3 00:01:37 2009 Sun May 10 00:00:39 2009 Sun May 17 00:00:42 2009 Sun May 24 00:00:53 2009 Sun May 31 00:00:50 2009 Sun Jun 7 00:01:04 2009 Sun Jun 14 00:00:55 2009 Sun Jun 21 00:01:17 2009 Sun Jun 28 00:01:10 2009 Sun Jul 5 00:03:09 2009 Sun Jul 12 00:01:10 2009 Sun Jul 19 00:01:18 2009 Sun Jul 26 00:01:52 2009 Sun Aug 2 00:01:27 2009 Sun Aug 9 00:02:39 2009 Sun Aug 16 00:01:52 2009 Sun Aug 23 00:01:33 2009 Sun Aug 30 00:01:42 2009 Sat Sep 5 17:23:02 2009 Sun Sep 6 00:00:41 2009 Sun Sep 13 00:00:47 2009 Sun Mar 7 00:04:47 2010 Sun Mar 14 00:07:12 2010 Sun Mar 21 00:01:36 2010 Sun Mar 28 00:02:14 2010