how to parse a very long file

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
jasongr
Forum Contributor
Posts: 206
Joined: Tue Jul 27, 2004 6:19 am

how to parse a very long file

Post by jasongr »

Hello

I have a textual log file which can be very very long
I need to write a PHP code that parses it, extracting information from it

Example includes:
1) The line in the file (terminating in "\n"
2) The number of times a specific phrase appears in the file
3) The latest lines in the file containing a specific phrase
4) The last 500 lines in the file

I know that I can do this by reading the file's content into memory and doing my own parsing, but this is not possible when the file gets very large (imagine a log file which is over 50MB)

Is there a way to invoke Linux commands from within my PHP application that will get the necessary information I need?

if so, I would appreciate it if someone could provide me with a code example, of how I can solve steps 1-4 using this mechanism (which I think would be much faster)

thanks in advance
User avatar
n00b Saibot
DevNet Resident
Posts: 1452
Joined: Fri Dec 24, 2004 2:59 am
Location: Lucknow, UP, India
Contact:

Post by n00b Saibot »

I am not a linux man but since you specifically ask me... I'd say looking into grep in more detail... it may solve all your problem at one go... :)
jasongr
Forum Contributor
Posts: 206
Joined: Tue Jul 27, 2004 6:19 am

Post by jasongr »

here is the solution that gave stereofrog me:

Code: Select all

1) The line in the file (terminating in "\n"

head -45 file.txt | tail -1 

prints 45th line

2) The number of times a specific phrase appears in the file

grep -c phrase file.txt

3) The latest lines in the file containing a specific phrase

grep phrase file.txt | tail

4) The last 500 lines in the file

tail -500 file.txt
Now I need to find out how one invokes Unix command from php
User avatar
twigletmac
Her Royal Site Adminness
Posts: 5371
Joined: Tue Apr 23, 2002 2:21 am
Location: Essex, UK

Post by twigletmac »

Good place to start: http://php.net/exec

Mac
jasongr
Forum Contributor
Posts: 206
Joined: Tue Jul 27, 2004 6:19 am

Post by jasongr »

thanks twigletmac, I will do just that
Post Reply