Linux command to transfer only the updated parts of files.

Whether you are using Linux on the desktop or as a server, it's still good that you're using Linux. Linux related questions go here.

Moderator: General Moderators

Post Reply
impulse()
Forum Regular
Posts: 748
Joined: Wed Aug 09, 2006 8:36 am
Location: Staffordshire, UK
Contact:

Linux command to transfer only the updated parts of files.

Post by impulse() »

I was reading a few months back of a Linux command that sends a file to a server and if the file exists already it only sends the parts of the file that need updating to bring it to it's latest state.

Could somebody point me in the right direction please?
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

Sounds similar to rsync.
impulse()
Forum Regular
Posts: 748
Joined: Wed Aug 09, 2006 8:36 am
Location: Staffordshire, UK
Contact:

Post by impulse() »

Thanking you very much.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Hmm, I didn't realise rsync did that :) Useful to know since I might need to replicate a code base around multiple servers soon, although until now I'd decided and using pure subversion.

Thanks :)
impulse()
Forum Regular
Posts: 748
Joined: Wed Aug 09, 2006 8:36 am
Location: Staffordshire, UK
Contact:

Post by impulse() »

I've had it working within the last 30 minutes and it's a brilliant piece of software. I've been planning to use it for months but only found good use for it this week.
impulse()
Forum Regular
Posts: 748
Joined: Wed Aug 09, 2006 8:36 am
Location: Staffordshire, UK
Contact:

Post by impulse() »

I've read the documentation that says RSYNC compresses the data before it's sent but what's happening at the moment is silly.

I've deleted the files that is creates as backups so RSYNC transfers the full file and it seems to be transferring a 14MB from a 144kbps UP connection in minutes. This used to take about 10-15 minutes with SCP but now it seems to do it in about 3-5 minutes.

I can confirm it's transferring the full 14MB file and not only missing bytes. The backup seems to untar OK when it's received so I can't see any problems with it.

Is this normal behavior for RSYNC?
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

Typically if the files are different, the newer file will override. Meaning the files aren't compared to see what's different, that would waste a lot of time.
User avatar
Ollie Saunders
DevNet Master
Posts: 3179
Joined: Tue May 24, 2005 6:01 pm
Location: UK

Post by Ollie Saunders »

feyd wrote:Meaning the files aren't compared to see what's different, that would waste a lot of time.
Well that depends on the size of the files now, doesn't it.

If you want to do that there's diff and patch but they only operate over the local filesystem.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

You could probably use subversion for this if they're not binary files.
impulse()
Forum Regular
Posts: 748
Joined: Wed Aug 09, 2006 8:36 am
Location: Staffordshire, UK
Contact:

Post by impulse() »

I thought Subversion would include some manual work.

I've setup SSH keys and dropped Rsync in a CRON job so it's completely automated, which is what I need.
I've even wrote a nifty little PHP script which will create a folder every Sunday night named 110707-180707 (or whatever the date is for 7 days before) and drop all current contents into there. Then start putting backups into the main directory again. I felt very smug when it all came together and worked :D
User avatar
Ollie Saunders
DevNet Master
Posts: 3179
Joined: Tue May 24, 2005 6:01 pm
Location: UK

Post by Ollie Saunders »

I felt very smug when it all came together and worked
I think I would too.
User avatar
volka
DevNet Evangelist
Posts: 8391
Joined: Tue May 07, 2002 9:48 am
Location: Berlin, ger

Post by volka »

feyd wrote:Typically if the files are different, the newer file will override. Meaning the files aren't compared to see what's different, that would waste a lot of time.
But isn't that a feature of rsync?
http://samba.anu.edu.au/rsync/features.html wrote:rsync uses the "rsync algorithm" which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

It's been years since I used it last. Honestly, I use Subversion hooks now. :)
User avatar
volka
DevNet Evangelist
Posts: 8391
Joined: Tue May 07, 2002 9:48 am
Location: Berlin, ger

Post by volka »

and I never cared about how it works ... it's the only part of a gentoo update that works without problems :)
Post Reply