Page 1 of 1

how to parse a very long file

Posted: Thu Nov 17, 2005 5:45 am
by jasongr
Hello

I have a textual log file which can be very very long
I need to write a PHP code that parses it, extracting information from it

Example includes:
1) The line in the file (terminating in "\n"
2) The number of times a specific phrase appears in the file
3) The latest lines in the file containing a specific phrase
4) The last 500 lines in the file

I know that I can do this by reading the file's content into memory and doing my own parsing, but this is not possible when the file gets very large (imagine a log file which is over 50MB)

Is there a way to invoke Linux commands from within my PHP application that will get the necessary information I need?

if so, I would appreciate it if someone could provide me with a code example, of how I can solve steps 1-4 using this mechanism (which I think would be much faster)

thanks in advance

Posted: Thu Nov 17, 2005 6:00 am
by n00b Saibot
I am not a linux man but since you specifically ask me... I'd say looking into grep in more detail... it may solve all your problem at one go... :)

Posted: Thu Nov 17, 2005 6:07 am
by jasongr
here is the solution that gave stereofrog me:

Code: Select all

1) The line in the file (terminating in "\n"

head -45 file.txt | tail -1 

prints 45th line

2) The number of times a specific phrase appears in the file

grep -c phrase file.txt

3) The latest lines in the file containing a specific phrase

grep phrase file.txt | tail

4) The last 500 lines in the file

tail -500 file.txt
Now I need to find out how one invokes Unix command from php

Posted: Thu Nov 17, 2005 6:19 am
by twigletmac
Good place to start: http://php.net/exec

Mac

Posted: Thu Nov 17, 2005 6:20 am
by jasongr
thanks twigletmac, I will do just that