Page 1 of 1

how to downlaod only new lines of a file via ftp

Posted: Thu Mar 10, 2011 12:17 am
by ummar1
i am downloading some log file (in which 100 lines are logged per sec) from ftp server via php.
every time i download the file download becomes slower as it size grows with time.
is there some way that i get only the new lines (added later) so that my download becomes faster.

Re: how to downlaod only new lines of a file via ftp

Posted: Thu Mar 10, 2011 6:51 am
by litebearer
8.6 million lines per day? Perhaps you should re-think how the log file is created/modified.
Maybe use datetime as the logfile name - ie 2011031000 would be from midnight til 00:59:59 ETC ETC

Re: how to downlaod only new lines of a file via ftp

Posted: Thu Mar 10, 2011 9:46 am
by pickle
Rotate your log - either on a time basis or file size basis.

Re: how to downlaod only new lines of a file via ftp

Posted: Thu Mar 10, 2011 10:43 am
by ummar1
thx for ur reply dear but i cannot alter my log file :(, i just have to download the file.
this file is generated by some other member and he just keep on doing his work and i have to do mine

Re: how to downlaod only new lines of a file via ftp

Posted: Thu Mar 10, 2011 10:49 am
by pickle
Then there's not really anything you can do. You should really tell the other member to rotate his logs though.

Re: how to downlaod only new lines of a file via ftp

Posted: Thu Mar 10, 2011 12:03 pm
by litebearer
How frequently do you need to download the log file?

Re: how to downlaod only new lines of a file via ftp

Posted: Thu Mar 10, 2011 9:43 pm
by ummar1
every sec

Re: how to downlaod only new lines of a file via ftp

Posted: Thu Mar 10, 2011 10:01 pm
by litebearer
seeing how it takes longer than a sec to download the log file, it seems you have multiple downloads going simultaneously?

Re: how to downlaod only new lines of a file via ftp

Posted: Thu Mar 10, 2011 10:13 pm
by jim.barrett
so you have a file that is growing at a rate of 100 lines per second and you need to upload it every second?? I really hope someone has an answer for this because I would love to hear it.

Re: how to downlaod only new lines of a file via ftp

Posted: Sat Mar 12, 2011 7:56 am
by ummar1
oh i have a simple solution :), i have written a simple c shell script

#!/bin/csh
while
do
tale -100 log.txt > downloadandandappend.txt
done

what this simple test do is it reads the last 100 lines of my file "log.txt" file (in linux) and then send it to a newly created file "downloadandandappend.txt " and then keeps on doing so
then i download this "downloadandandappend.txt " which always recreates on every cycle of while statement.

so i have the solution but for this i have to run the script on linux side first, more efficient solution is that if i can run this shell script from php. i know the function "shell_exec" but it says that i cannot run this, php is not in safe mode.
how can i run this shell script command help me please.
i am using php designer version 7.2.2 and wamp server