how to downlaod only new lines of a file via ftp

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
ummar1
Forum Newbie
Posts: 4
Joined: Thu Mar 10, 2011 12:13 am

how to downlaod only new lines of a file via ftp

Post by ummar1 »

i am downloading some log file (in which 100 lines are logged per sec) from ftp server via php.
every time i download the file download becomes slower as it size grows with time.
is there some way that i get only the new lines (added later) so that my download becomes faster.
litebearer
Forum Contributor
Posts: 194
Joined: Sat Mar 27, 2004 5:54 am

Re: how to downlaod only new lines of a file via ftp

Post by litebearer »

8.6 million lines per day? Perhaps you should re-think how the log file is created/modified.
Maybe use datetime as the logfile name - ie 2011031000 would be from midnight til 00:59:59 ETC ETC
User avatar
pickle
Briney Mod
Posts: 6445
Joined: Mon Jan 19, 2004 6:11 pm
Location: 53.01N x 112.48W
Contact:

Re: how to downlaod only new lines of a file via ftp

Post by pickle »

Rotate your log - either on a time basis or file size basis.
Real programmers don't comment their code. If it was hard to write, it should be hard to understand.
ummar1
Forum Newbie
Posts: 4
Joined: Thu Mar 10, 2011 12:13 am

Re: how to downlaod only new lines of a file via ftp

Post by ummar1 »

thx for ur reply dear but i cannot alter my log file :(, i just have to download the file.
this file is generated by some other member and he just keep on doing his work and i have to do mine
User avatar
pickle
Briney Mod
Posts: 6445
Joined: Mon Jan 19, 2004 6:11 pm
Location: 53.01N x 112.48W
Contact:

Re: how to downlaod only new lines of a file via ftp

Post by pickle »

Then there's not really anything you can do. You should really tell the other member to rotate his logs though.
Real programmers don't comment their code. If it was hard to write, it should be hard to understand.
litebearer
Forum Contributor
Posts: 194
Joined: Sat Mar 27, 2004 5:54 am

Re: how to downlaod only new lines of a file via ftp

Post by litebearer »

How frequently do you need to download the log file?
ummar1
Forum Newbie
Posts: 4
Joined: Thu Mar 10, 2011 12:13 am

Re: how to downlaod only new lines of a file via ftp

Post by ummar1 »

every sec
litebearer
Forum Contributor
Posts: 194
Joined: Sat Mar 27, 2004 5:54 am

Re: how to downlaod only new lines of a file via ftp

Post by litebearer »

seeing how it takes longer than a sec to download the log file, it seems you have multiple downloads going simultaneously?
jim.barrett
Forum Newbie
Posts: 20
Joined: Tue Mar 01, 2011 5:47 am

Re: how to downlaod only new lines of a file via ftp

Post by jim.barrett »

so you have a file that is growing at a rate of 100 lines per second and you need to upload it every second?? I really hope someone has an answer for this because I would love to hear it.
ummar1
Forum Newbie
Posts: 4
Joined: Thu Mar 10, 2011 12:13 am

Re: how to downlaod only new lines of a file via ftp

Post by ummar1 »

oh i have a simple solution :), i have written a simple c shell script

#!/bin/csh
while
do
tale -100 log.txt > downloadandandappend.txt
done

what this simple test do is it reads the last 100 lines of my file "log.txt" file (in linux) and then send it to a newly created file "downloadandandappend.txt " and then keeps on doing so
then i download this "downloadandandappend.txt " which always recreates on every cycle of while statement.

so i have the solution but for this i have to run the script on linux side first, more efficient solution is that if i can run this shell script from php. i know the function "shell_exec" but it says that i cannot run this, php is not in safe mode.
how can i run this shell script command help me please.
i am using php designer version 7.2.2 and wamp server
Post Reply