Page 1 of 1
Linux command to transfer only the updated parts of files.
Posted: Mon Jul 09, 2007 2:29 pm
by impulse()
I was reading a few months back of a Linux command that sends a file to a server and if the file exists already it only sends the parts of the file that need updating to bring it to it's latest state.
Could somebody point me in the right direction please?
Posted: Mon Jul 09, 2007 2:51 pm
by feyd
Sounds similar to rsync.
Posted: Mon Jul 09, 2007 3:03 pm
by impulse()
Thanking you very much.
Posted: Mon Jul 09, 2007 4:05 pm
by Chris Corbyn
Hmm, I didn't realise rsync did that

Useful to know since I might need to replicate a code base around multiple servers soon, although until now I'd decided and using pure subversion.
Thanks

Posted: Mon Jul 09, 2007 4:47 pm
by impulse()
I've had it working within the last 30 minutes and it's a brilliant piece of software. I've been planning to use it for months but only found good use for it this week.
Posted: Tue Jul 10, 2007 9:43 am
by impulse()
I've read the documentation that says RSYNC compresses the data before it's sent but what's happening at the moment is silly.
I've deleted the files that is creates as backups so RSYNC transfers the full file and it seems to be transferring a 14MB from a 144kbps UP connection in minutes. This used to take about 10-15 minutes with SCP but now it seems to do it in about 3-5 minutes.
I can confirm it's transferring the full 14MB file and not only missing bytes. The backup seems to untar OK when it's received so I can't see any problems with it.
Is this normal behavior for RSYNC?
Posted: Tue Jul 10, 2007 2:50 pm
by feyd
Typically if the files are different, the newer file will override. Meaning the files aren't compared to see what's different, that would waste a lot of time.
Posted: Tue Jul 10, 2007 3:20 pm
by Ollie Saunders
feyd wrote:Meaning the files aren't compared to see what's different, that would waste a lot of time.
Well that depends on the size of the files now, doesn't it.
If you want to do that there's diff and patch but they only operate over the local filesystem.
Posted: Tue Jul 10, 2007 3:56 pm
by Chris Corbyn
You could probably use subversion for this if they're not binary files.
Posted: Wed Jul 11, 2007 6:18 am
by impulse()
I thought Subversion would include some manual work.
I've setup SSH keys and dropped Rsync in a CRON job so it's completely automated, which is what I need.
I've even wrote a nifty little PHP script which will create a folder every Sunday night named 110707-180707 (or whatever the date is for 7 days before) and drop all current contents into there. Then start putting backups into the main directory again. I felt very smug when it all came together and worked

Posted: Wed Jul 11, 2007 6:38 am
by Ollie Saunders
I felt very smug when it all came together and worked
I think I would too.
Posted: Fri Jul 13, 2007 7:32 am
by volka
feyd wrote:Typically if the files are different, the newer file will override. Meaning the files aren't compared to see what's different, that would waste a lot of time.
But isn't that a feature of rsync?
http://samba.anu.edu.au/rsync/features.html wrote:rsync uses the "rsync algorithm" which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand.
Posted: Fri Jul 13, 2007 7:36 am
by feyd
It's been years since I used it last. Honestly, I use Subversion hooks now.

Posted: Fri Jul 13, 2007 7:59 am
by volka
and I never cared about how it works ... it's the only part of a gentoo update that works without problems
