PHP script crashing apache after a few hours...

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
jclarkkent2003
Forum Contributor
Posts: 123
Joined: Sat Dec 04, 2004 9:14 pm

PHP script crashing apache after a few hours...

Post by jclarkkent2003 »

Hi again fellas, wazzup?

lol... I've been coding like a fr3@k lately, non stop, and well, here's my dilemma of the day.....

I have this script that parses a very large file, and it's got plenty of them, after it finishes one ( takes maybe 20-30 minutes each ) it goes to the next one. Problem is, after it's running for a few hours, apache crashes.

I have been wanting to get it translated into c and run it in the background 24/7, but i don't know enough c and the people at vbforums are NOT very helpful, lol, posted a thread last week and been viewed only 13 times no replies......

So what can I do to make this php script more efficient and or not crash, or break it up into multiples......

Like would it have any effect if I cleaned unset() all the variables at the end of each for loop or would that not matter, since at the beginning of each for loop all the variables are set equalt to default ?


Also, is there a possible way to say, I make the php script only run ONCE, can I multithread it's foor loop or is PHP a single threading language where c is multithreaded ?

I need to give it a base of like 100-1000 LARGE root files to parse and shred each day, and basically it's going to take ALL day or longer to do that, maybe it only gets a few hundred files done a day maybe more or less.....

I need to find a way that it keeps going, like if apache crashes after 2 hours, and it only finished 5-10 files, it's not going to do anything for the rest of the time. How can I make it so I have MULTIPLE instances of it running (in the background of course) or something of that nature? \

I've been trying to figure this out on my own for the past week or two but I can't quite see the alog to code in php and what will run it , cron ? I mean I was thinking to integrate cron to run the php script every so often, but i'm not sure how well that will work, and of course, LIST.txt is the LIST of 100-1000 large files it's got to parse and do it's trick to...

Thanks in advance!
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

If you have shell access, just bypass Apache and run it as a CLI script.
jclarkkent2003
Forum Contributor
Posts: 123
Joined: Sat Dec 04, 2004 9:14 pm

Post by jclarkkent2003 »

that's interesting, I thought even when you ran it as command line, that it was using apache, i'm glad it's not.

Do you know WHICH ip address it will show up as, if for instance my script went on the internet for some functions i have in there, it verifies some stuff and grabs some stuff, etc...

Do you know how I would SET which ip address it would use? Example, i have 60 ip's assigned and binded on my server, and i'd like to make my script rotate through using them, so it doesn't show one server hitting the net evrey time, but rather 60 different IP's ?

Yes, of course I have shell access, lol, I'm guessing if I ran it as shell, that I would still do the

set_time_limit(0); and ignore_user_abort(1);

and it would run forever, without crashing? Any advice on clearing the cache or ram or such like that, so anything that was included or "loaded" in the for loop, would be wiped 100% clean or free'd up before the next run? I don't think ob_start() and ob_flush(), ob_end_clean() is the proper way to do that, I tried and it didn't work correctly.

Multithreading ? Possible in a for loop with php or not possible? ( not sure if i learned that php is a single threaded language or not )....

Thanks.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Post by josh »

nice -n 19 ./script.php > /dev/null 2>&1 &

Will let your script run even after you log out of SSH.

We can't tell you if your script has memory problems without seeing it.

By the way, the nice command sets it to the lowest priority so it doesn't get all the CPU attention (since you'll have it running 24/7)
jclarkkent2003
Forum Contributor
Posts: 123
Joined: Sat Dec 04, 2004 9:14 pm

Post by jclarkkent2003 »

k thanks, i like that nice command :)


> Do you know WHICH ip address it will show up as, if for instance my script went on the internet for some functions i have in there, it verifies some stuff and grabs some stuff, etc...

> Do you know how I would SET which ip address it would use? Example, i have 60 ip's assigned and binded on my server, and i'd like to make my script rotate through using them, so it doesn't show one server hitting the net evrey time, but rather 60 different IP's ?

> Multithreading ? Possible in a for loop with php or not possible? ( not sure if i learned that php is a single threaded language or not )....

Got any info on these two ? :) The IP thing is very important and multithreading would be awesome!
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Post by josh »

Do you know WHICH ip address it will show up as, if for instance my script went on the internet for some functions i have in there, it verifies some stuff and grabs some stuff, etc...
Why don't you try it?
Do you know how I would SET which ip address it would use? Example, i have 60 ip's assigned and binded on my server, and i'd like to make my script rotate through using them, so it doesn't show one server hitting the net evrey time, but rather 60 different IP's ?
No idea, my guess is you might have to set up an ipchains rule on linux
Multithreading ? Possible in a for loop with php or not possible? ( not sure if i learned that php is a single threaded language or not )....
no, php is not multi-threaded. your best bet would be a different language or break up whatever you're doing into chunks of work to have multiple instance of your script at work
Got any info on these two ? :) The IP thing is very important and multithreading would be awesome!
That was three :) For some reason the IP thing sounds black-hat, is there an actual legit reason why they can't see one of your ips more than once? I see no reason unless you are:
  • not obeying robots.txt
  • hammering their server
  • bruteforcing something / running some kind of exploit
  • Stealing content
please correct me if I am wrong, I sure hope I am for those webmaster's sakes[/quote]
jclarkkent2003
Forum Contributor
Posts: 123
Joined: Sat Dec 04, 2004 9:14 pm

Post by jclarkkent2003 »

k i guess i hope there is someone else here who knows how to do that ipchains rules or whatever, lol... I never mess with that, all I know about ipchains is it's a firewall or that's what its most commonly used as....

So anyone Else know how to use all 60 of the ips binded to my server with command line shell php ?
Post Reply