Why would I use bup and not rsync/rsnapshot?

Hey,

bup provides botz deduplication and compression.

Say you have 2 Debian servers. both of them have a bunch of data in
common. (This can be a media library on many computers, youre email or
whatever).

Now you backup the first server. bup's magic happens. files are split up
in chunks, those are packed together in packfiles, nothing really to
bother you. If you want to know the "magic" behind it please read the
DESIGN document or ask for specific explanations.

When finished you start a backup on the second server. Here again files
are split up. Duplicate chunks aren't saved a second time, they are just
referenced. This is what deduplaction is.

bup uses a algorithm similar to the one that rsnapshot uses for
efficient transmissions. bup uses it for deduplication.

VM images is where bup shines and more or less was designed for: You
have huge files that change a little.

I'm sorry but don't have to much time at the moment, please skim through
the DESIGN document to read what bup does.

I'll have a look at your benchmarking tool ASAP to see how we can
possibly use it to benchmark bup as well.

I did some "benchmarks" some time ago [1]. I imported my rsnapshot
backups to bup.

TL;TR:
rsnapshot: 12.6G
bup: 4.6G

> Dieter

Thanks for your interest!
Zoran
>
> [*] http://dieter.plaetinck.be/rsyncbench_an_rsync_benchmarking_tool

[1]
http://groups.google.com/group/bup-list/browse_thread/thread/8a426b233554670/e67926cfd69900fb?lnk=gst&q=importing#e67926cfd69900fb

Except in the case of file renames/moves. Bup
1. saves backup space,
2. saves transfer time, and
3. keeps old snapshots as well.

With rsync transfer time is greater, and backup space is greater unless old
snapshots are removed.

just my quick thoughts:

two servers get backed up to another one. I'll do it with Amazon EC2
instances. I'll measure the traffic using NIC counters.

I'll do 3 runs:

Initial backup

a second without changed data

a third with some added data. I'll try to mimic a changing VM Image by
generating a 1G textfile and changing some lines somewhere in the middle.

I'll do all of this for two servers (with different fake VM images) for
both bup and rsnapshot.

I'll measure:
* backup time
* transfered data
* backup space

Any thoughts on that?

Zoran

Allright I'll do so.

My Upstream connection isn't to good, so uploading some VM images isn't
an option.

What do you mean "not 100% accurate"? There's no reason I can imagine
that the counters on eth0/eth1/etc shouldn't be anything but accurate.

The down side is that you'll get things like TCP headers and
retransmits thrown into your count, as well as traffic on any other
ports you use at the time. You might argue that those are in fact
*more* accurate than not including them, since if (say) bup sent a
whole bunch of one-byte packets, you're technically paying for more
TCP headers. Of course, neither bup nor rsnapshot nor rsync do
anything like that so it doesn't matter.

Beware that tcpdump can also drop packets sometimes, though.

When I'm testing stuff, I usually use iptables accounting rules.

iptables -A OUTPUT -p tcp --port 22 -j ACCEPT
...
iptables -nvL OUTPUT

The rule you added should then have counters for how many bytes
matched that rule. (Note: I haven't tested the above commands, so
there might be a typo or two.)

Have fun,

Avery

# 1. run - first backup
## bup
time: 219 s
transferred: 165417,61 KB
disk space: 156804 KB

## rsnapshot
time: 60s
transferred: 888294,84 KB
953708 KB

# 2. run
## bup
time: 0 s
transferred: 21,65 KB
disk space: 156820 KB

## rsnapshot
time: 4 s
transferred: 1270,8 KB
disk space: 968784 KB

# 3. run - generated a 1G fake-image with data from /dev/urandom on each
## bup
time: 691 s
transferred: 2208538,07 KB
disk space: 2277004 KB

## rsnapshot
time: 133 s
transferred: 2190208,71 KB
disk space: 3083116 KB

# 4. run
## bup
time: 12 s
transferred: 21,99 KB
disk space: 2281276 KB

## rsnapshot
time: 4 s
transferred: 1239,87 KB
disk space: 3098200 KB

# 5. run - changed 1M in the middle of the fake image file
## bup
time: 106 s
transferred: 2249,64 KB
disk space: 2281920 KB

## rsnapshot
time: 119 s
transferred: 3696,62 KB
disk space: 5212580 KB

# total
## bup
time: 1037 s
transferred: 2376248,96 KB
disk space: 2281920 KB

## rsnapshot
time: 320 s
transferred: 3084710,84
disk space: 5212580 KB

If anyone has questions feel free to ask.

Zoran

Man, bup is slow. Definitely need to work on that :)

Thanks for all your work!

Have fun,

Avery

I suppose that this does not really show bup's advantage when backing up
multiple similar systems and moving/renaming big files or directories between
snapshots.

Well, two instances of the same image are pretty much similar systems.

Sure I could have copied and/or moved around my fake images, transferred
around between the server, but I think it's pretty clear where bup
shines. Take a look at the rest of the "benchmark".

I think disk space- and transfer-wise looks pretty good.

What would your improvements be?

Zoran

On the other hand, by the time you've obtained a 9x performance
increase, adding file renames into that might be just bragging :)

Have fun,

Avery

I wrote a import-rsnapshot command, which just needs some testcases
before I submit the patches. I pushed it to my github repo:

http://github.com/zoranzaric/bup/tree/import-rsnapshot

Feel free to test and use it. It should make the transition from
rsnapshot to bup pretty easy.

Keep in mind that bup's master (and the import-rsnapshot branch) don't
save metadata like permissions, yet.

Rob is working on it and it'll be great.

Zoran

Exactly.

Zoran

You simply can't compress randomness; try it sometime. If you could
compress it, it wouldn't be random.

He listed the commands; it's obvious from those what he means.

The trends persist across any sort of files. Other than the
improvements over rsnapshot that come from renaming, of course, since
presumably you aren't renaming your VM images. As far as bup is
concerned, renames are only the change of a few bytes (the filename)
and not anything else.

To paraphrase: Zoran backed up the contents of his VM images, while
you plan to back up the raw VM disk files using your host system.

I'm not quite sure why you don't just test it on your own data; it
will only take a few minutes to set up, and then you'll have the final
answer on *your* data, not a synthetic benchmark.

But anyway, I expect that you'll find the results quite excellent. I
certainly have when I've backed up my VM images. VM disks tend to
have a lot of duplication outside the gzip compression window (about
32-128k) because when you copy a file around and then delete the
original, you end up with chunks of the same file in two totally
different places on the disk. gzip fails badly at compressing such
things - it only really fixes redundancy inside that small window -
and so bup usually compresses better than gzip on even a *single* copy
of a VM disk image, if that VM disk has been busy in the past.

rsnapshot just stores (I think gzipped?) copies of the VM disk, so the
disk space usage of bup should be much less.

As for file transfer, I'd expect bup and rsnapshot to be pretty close
to each other - for the first backup. After that, bup retains
persistent state on the client side (the .idx and .midx files) which
will allow it to do future backups while sending far fewer bytes. And
of course you won't have to store the entire file over again like you
would with rsnapshot.

Basically: just try it. It's better.

(Except for speed. It's getting to be time to rewrite more of bup in
C, I guess. :))

Have fun,

Avery

yeah bup should work fine and restoring works ;)

I'm sorry if something i wrote wasn't clear. I'm not a native speaker,
so please excuse my inaccuracies.

Zoran

Starting with a pack indexer? :D

I'm limping along manually indexing packs on my big backup, which is
mostly OK, since I'm not making really big changes there, just a pack or
two now and then.

-Z

As a matter of fact, the pack indexer would probably be plenty fast
even in python :) But you're right, it does need to be written. I
happen to be on a road trip right now that's interfering with such
things, but I promise it's on my list.

Or if someone else around here is feeling motivated, it's not actually
too hard. Just look for the place where we call 'git index-pack' and
replace it :)

Have fun,

Avery

In my case I was backing up a remote $HOME to a low powered NAS. Hard
link trees are simply not an appropriate solution. I think by the time
I gave up on rsnapshot, I had more space used in filesystem overhead
than files themselves.