Tux

...making Linux just a little more fun!

rsync options

Martin J Hooper [martinjh at blueyonder.co.uk]


Wed, 05 Dec 2007 08:29:31 +0000

Just a quick question guys...

Backed up my Windows Document directory using rsync in Ubuntu with the following command line:

rsync -Havcx --progress --stats /home/martin/win/My\ Documents/*
/home/martin/back/mydocs/
(Mounted ntfs drive to mounted smbfs share)

If I run that again a few weeks or so later will it copy all 31000+ files again or will it just copy new and changed files?


Top    Back


Kapil Hari Paranjape [kapil at imsc.res.in]


Wed, 5 Dec 2007 14:07:55 +0530

Hello,

On Wed, 05 Dec 2007, Martin J Hooper wrote:

> Backed up my Windows Document directory using rsync in Ubuntu
> with the following command line:
> 
> rsync -Havcx --progress --stats /home/martin/win/My\ Documents/*
> /home/martin/back/mydocs/
> 
> (Mounted ntfs drive to mounted smbfs share)

I didn't know that ntfs supports hard links (-H flag). Live and learn, I suppose.

> If I run that again a few weeks or so later will it copy all
> 31000+ files again or will it just copy new and changed files?

It will copy only changes. Even for individual files it will only copy changed blocks.

Regards,

Kapil. --


Top    Back


Martin J Hooper [martinjh at blueyonder.co.uk]


Wed, 05 Dec 2007 08:51:23 +0000

Kapil Hari Paranjape wrote:

> Hello,
> I didn't know that ntfs supports hard links (-H flag). Live and
> learn, I suppose. 

It seemed to work ok so it might do.. ;) Not sure.

>> If I run that again a few weeks or so later will it copy all
>> 31000+ files again or will it just copy new and changed files?
> 
> It will copy only changes. Even for individual files it will only copy
> changed blocks.
> 

Brilliant! Cheers. Anyone suggest a better command line or will that one do?


Top    Back


Kapil Hari Paranjape [kapil at imsc.res.in]


Wed, 5 Dec 2007 14:31:53 +0530

Hello,

On Wed, 05 Dec 2007, Martin J Hooper wrote:

> Kapil Hari Paranjape wrote:
> >It will copy only changes. Even for individual files it will only copy
> >changed blocks.
> 
> Brilliant!  Cheers.  Anyone suggest a better command line or will 
> that one do?

The one you use seems fine.

I should have said that "rsync" will still check each file in the file-system for changes. This is because it is a history-less system.

An alternative that keeps some history is "unison". As a result the latter is often quicker if there are very few changed files but the volume is very large.

Regards,

Kapil. --


Top    Back


Jim Jackson [jj at franjam.org.uk]


Wed, 5 Dec 2007 10:36:56 +0000 (GMT)

On Wed, 5 Dec 2007, Martin J Hooper wrote:

> Just a quick question guys...
>
> Backed up my Windows Document directory using rsync in Ubuntu
> with the following command line:
>
> rsync -Havcx --progress --stats /home/martin/win/My\ Documents/*
> /home/martin/back/mydocs/

I also use the -u option "update only (don?t overwrite newer files)". You may wish to check out the --delete option, and --delete-after, depending on what you want. Otherwise deleted files will hang around on your backup. But YMMV.


Top    Back


Kapil Hari Paranjape [kapil at imsc.res.in]


Wed, 5 Dec 2007 16:25:29 +0530

Hello,

On Wed, 05 Dec 2007, Kapil Hari Paranjape wrote:

> On Wed, 05 Dec 2007, Martin J Hooper wrote:
> > Brilliant!  Cheers.  Anyone suggest a better command line or will 
> > that one do?
> 
> The one you use seems fine.

If your target supports hard links then a different way to use "rsync" is to have a date based archive.

	TODAY=$(date +%d-%m-%y)
	YEST=$(date +%d-%m-%y -d 'yesterday')
  
	rsync -Havx --link-dest=target/$YEST source target/$TODAY
This will create hard links under target/$TODAY for all files that already exist under target/$YEST. Any newer or changed files will be copied in the usual "rsync-way".

WARNING: This creates the entire directory tree anew and so will take longer than the usual procedure.

Regards,

Kapil. --


Top    Back


Ben Okopnik [ben at linuxgazette.net]


Wed, 5 Dec 2007 09:31:41 -0500

On Wed, Dec 05, 2007 at 08:29:31AM +0000, Martin J Hooper wrote:

> Just a quick question guys...
> 
> Backed up my Windows Document directory using rsync in Ubuntu
> with the following command line:
> 
> rsync -Havcx --progress --stats /home/martin/win/My\ Documents/*
> /home/martin/back/mydocs/
> 
> (Mounted ntfs drive to mounted smbfs share)
> 
> If I run that again a few weeks or so later will it copy all
> 31000+ files again or will it just copy new and changed files?

Just the diffs - as the 'rsync' man page specifies - and even "more so" due to your '-c' option:

 -c, --checksum              skip based on checksum, not mod-time & size
-- 
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *

Top    Back


Ben Okopnik [ben at linuxgazette.net]


Wed, 5 Dec 2007 09:38:36 -0500

On Wed, Dec 05, 2007 at 08:51:23AM +0000, Martin J Hooper wrote:

> Kapil Hari Paranjape wrote:
> > Hello,
> 
> > I didn't know that ntfs supports hard links (-H flag). Live and
> > learn, I suppose. 
> 
> It seemed to work ok so it might do.. ;)  Not sure.
> 
> >> If I run that again a few weeks or so later will it copy all
> >> 31000+ files again or will it just copy new and changed files?
> > 
> > It will copy only changes. Even for individual files it will only copy
> > changed blocks.
> > 
> 
> Brilliant!  Cheers.  Anyone suggest a better command line or will 
> that one do?

Well, you might want to review all the options you're using in the 'rsync' man page. As Kapil said, what you're using seems fine - but that's only because we're assuming that you want rsync to do the "average" thing (a bit of circular reasoning, there. :) Only you can tell if the options actually match what you want to do.

-- 
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *

Top    Back


Martin J Hooper [martinjh at blueyonder.co.uk]


Wed, 05 Dec 2007 15:24:29 +0000

Ben Okopnik wrote:

>  -c, --checksum              skip based on checksum, not mod-time & size

That might be the reason it takes ages to work out at the beginning how many files its going to copy...

All I want to do Ben is take a complete copy of my Docs making sure that the directory structure is the same.


Top    Back


Ben Okopnik [ben at linuxgazette.net]


Wed, 5 Dec 2007 11:54:37 -0500

On Wed, Dec 05, 2007 at 03:24:29PM +0000, Martin J Hooper wrote:

> Ben Okopnik wrote:
> >  -c, --checksum              skip based on checksum, not mod-time & size
> 
> That might be the reason it takes ages to work out at the 
> beginning how many files its going to copy...
> 
> All I want to do Ben is take a complete copy of my Docs making 
> sure that the directory structure is the same.

Y'know, I often hear people bitching about how unfrienldly man pages are [1] - but the one for "rsync" is very, very friendly indeed. [2] It doesn't just explain the options - it also shows how to use the prog in the most common, useful ways. The way I typically use "rsync" comes straight from that page (and yes, I have taken the time to read up on the options that I use - I hate cargo cult coding, and assiduously avoid planting landmines in my own path.)

So, to finally answer your question [grin]:

# Note: you don't need the '*' at the end of the source spec as long as
# you have that '/'. You also don't need to copy 'H'ard links: AFAIK,
# Wind0ws doesn't have those. You also don't need the 'x' (unless your
# Documents dir crosses filesystems!) I'll assume that you want to see
# that progress bar and those stats...
  
rsync -avz --progress --stats ~/win/My\ Documents/ ~/back/mydocs/
In other words, '-avz' does the greatest majority of what people usually want in that situation.

[1] "UNIX is user friendly. It's just selective about who its friends are."

[2] As pTerry said, "...sir, to forestall your next question, I personally would go to bed with it."

-- 
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *

Top    Back


Martin J Hooper [martinjh at blueyonder.co.uk]


Wed, 05 Dec 2007 19:07:38 +0000

Ben Okopnik wrote:

> Y'know, I often hear people bitching about how unfrienldly man
> pages are [1] - but the one for "rsync" is very, very friendly
> indeed. [2] It doesn't just explain the options - it also
> shows how to use the prog in the most common, useful ways. The
> way I typically use "rsync" comes straight from that page (and
> yes, I have taken the time to read up on the options that I
> use - I hate cargo cult coding, and assiduously avoid planting
> landmines in my own path.)

Does any one know of a good GUI manpage reader for Gnome?

In KDE you could just type man:/rsync into a Konq window and a nicely HTML formatted man page would pop up and I just wondered if you could so the same thing in Gnome...


Top    Back


Jim Jackson [jj at franjam.org.uk]


Wed, 5 Dec 2007 19:17:35 +0000 (GMT)

On Wed, 5 Dec 2007, Martin J Hooper wrote:

> Ben Okopnik wrote:
> > Y'know, I often hear people bitching about how unfrienldly man
> > pages are [1] - but the one for "rsync" is very, very friendly
> > indeed. [2] It doesn't just explain the options - it also
> > shows how to use the prog in the most common, useful ways. The
> > way I typically use "rsync" comes straight from that page (and
> > yes, I have taken the time to read up on the options that I
> > use - I hate cargo cult coding, and assiduously avoid planting
> > landmines in my own path.)
>
> Does any one know of a good GUI manpage reader for Gnome?
>
> In KDE you could just type man:/rsync into a Konq window and a
> nicely HTML formatted man page would pop up and I just wondered
> if you could so the same thing in Gnome...

seriously - why?

The search facilities under less (the usual manpager) are way superior and quicker than anything mouse driven. And for long man pages it is searching that is the major power tool.


Top    Back


Martin J Hooper [martinjh at blueyonder.co.uk]


Wed, 05 Dec 2007 19:28:23 +0000

Jim Jackson wrote:

> seriously - why?
> 
> The search facilities under less (the usual manpager) are way superior and
> quicker than anything mouse driven. And for long man pages it is searching
> that is the major power tool.

To be honest I prefer GUI text for reading something long as I prefer the text on the GUI to the ternimal...


Top    Back


Jim Jackson [jj at franjam.org.uk]


Wed, 5 Dec 2007 20:03:09 +0000 (GMT)

On Wed, 5 Dec 2007, Martin J Hooper wrote:

> Jim Jackson wrote:
> > seriously - why?
> >
> > The search facilities under less (the usual manpager) are way superior and
> > quicker than anything mouse driven. And for long man pages it is searching
> > that is the major power tool.
>
> To be honest I prefer GUI text for reading something long as I
> prefer the text on the GUI to the ternimal...

ah - form over functionality.

Fashion wins every time :-)


Top    Back


Rick Moen [rick at linuxmafia.com]


Wed, 5 Dec 2007 12:12:29 -0800

Quoting Martin J Hooper (martinjh@blueyonder.co.uk):

> Does any one know of a good GUI manpage reader for Gnome?

/usr/bin/man in an xterm.

-- 
Cheers,                                     Ceci n'est pas une pipe:   |
Rick Moen
rick@linuxmafia.com 

Top    Back


Ben Okopnik [ben at linuxgazette.net]


Wed, 5 Dec 2007 15:37:38 -0500

On Wed, Dec 05, 2007 at 12:12:29PM -0800, Rick Moen wrote:

> Quoting Martin J Hooper (martinjh@blueyonder.co.uk):
> 
> > Does any one know of a good GUI manpage reader for Gnome?
> 
> /usr/bin/man in an xterm.

Damn, Rick, you beat me to it. :) "xterm -e man xterm" would be my version - especially since you can tweak "xterm" to be as pretty as any GUI out there.

-- 
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *

Top    Back


Thomas Adam [thomas.adam22 at gmail.com]


Wed, 5 Dec 2007 21:07:01 +0000

On 05/12/2007, Ben Okopnik <ben@linuxgazette.net> wrote:

> On Wed, Dec 05, 2007 at 12:12:29PM -0800, Rick Moen wrote:
> > Quoting Martin J Hooper (martinjh@blueyonder.co.uk):
> >
> > > Does any one know of a good GUI manpage reader for Gnome?
> >
> > /usr/bin/man in an xterm.
>
> Damn, Rick, you beat me to it. :) "xterm -e man xterm" would be my
> version - especially since you can tweak "xterm" to be as pretty as any
> GUI out there.

There's always tkman.

-- Thomas Adam


Top    Back