Help! Runaway Crawler!!

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
beserker
Forum Newbie
Posts: 5
Joined: Fri Oct 08, 2004 8:04 am

Help! Runaway Crawler!!

Post by beserker »

I've just been developing a simple web crawler today, but it went "off-piste" and started crawling the entire web (well it got as far as Google before I pressed 'stop' in my browser).

The thing is I'm worried that the script's still running even though I did press stop - can anyone clarify this?

The setup is this - the script basically starts at a certain URL, get's the links out of that page then follows them one by one getting more links etc. etc. - each time it finds a new link it print()'s it to the browser - so I sit there watching the links appear as the script runs. The core component of it is a recursive loop so I'm worried that it'll never stop...
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

Check the processes on the machine. If PHP is actively running (and likely eating the processor) chances are, it's still running. Get or use your operating system's favored process killing program to kill it if it's still running.
beserker
Forum Newbie
Posts: 5
Joined: Fri Oct 08, 2004 8:04 am

Post by beserker »

That's it - I can't - it's on a remote host (shared hosting) and the support guy said that he "couldn't stop PHP" and that I had to send a support email in... brilliant.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

So.. do it. :?
beserker
Forum Newbie
Posts: 5
Joined: Fri Oct 08, 2004 8:04 am

Post by beserker »

I have... no reply yet... but what I was wondering about was whether the script would keep executing after pressing 'stop' in the browser.
User avatar
John Cartwright
Site Admin
Posts: 11470
Joined: Tue Dec 23, 2003 2:10 am
Location: Toronto
Contact:

Post by John Cartwright »

Possibly but likely it has timed out, depending on your configuration for ignore user about and maximum execution
beserker
Forum Newbie
Posts: 5
Joined: Fri Oct 08, 2004 8:04 am

Post by beserker »

Thanks - I spose the maximum execution time would have cut it off anyway then - I hadn't heard of that before

What's that ignore user about thing though? I couldn't see anything like that in php.ini
Post Reply