Page 1 of 1

Problem with long running script

Posted: Thu Apr 01, 2010 10:41 am
by sailorjohn
I've written a script that is supposed to execute a number of long running processes. Trouble is, it stops.

This is for a job board, written in PHP with MySQL as a database, running on a Linux box. The processes are set up as functions. I'm using Lynx to run the script as a cron job.

When the main script stops, I need to complete the jobs by running separate scripts from my desktop using a regular browser like Firefox or IE. As the script stops early on in the process, I spend a good deal of time running subsequent step "manually", one at a time.

Here are the steps:

1. Delete the old jobs2 table, create a new one, and insert data from jobs to jobs 2. No problems here.

2. Download xml data feeds (jobs) to the hard drive. There are about 165 feeds now, of varying sizes, some pretty big. This usually stops executing after a few records, and I have to finish it manually.

From this point on, I am usually running the individual scripts one at a time to complete the job.

3. Read the xml data feeds and insert into the jobs2 table. There are currently about 500,000 records. This stops, and needs to be restarted at least once.

Other steps seem to run OK, so I guess it's really the first two.

I should mention that I'm using full text indexing, so it's pretty intensive, but when I run top or mtop, everything seems OK.

I've got set_time_limit(0); so timeout is (I imagine) not the problem.

Any clues on how I can track this down?

Thanks in advance.

John

Re: Problem with long running script

Posted: Thu Apr 01, 2010 11:10 am
by M2tM
Check the ini variable max_input_time as well. There is a bug that can cause certain versions to use that instead of max_execution_time for script execution:

http://bugs.php.net/37306

Re: Problem with long running script

Posted: Thu Apr 01, 2010 12:05 pm
by Benjamin
I would also check the error log. You may be hitting memory limits.

Re: Problem with long running script

Posted: Thu Apr 01, 2010 7:33 pm
by sailorjohn
Thanks for the input. I boosted max_input_time and max_execution_time, set mysql connect_time = -1 (to turn it off) and also set auto_detect_line_endings=On just to cover that. I'll see if it runs tonight.

I haven't found anything in the logs to indicate a problem, which is kind of annoying. I've checked the Apache error log, plus messages. Any place else I should look?

Thanks.

John

Re: Problem with long running script

Posted: Sun Apr 04, 2010 7:24 pm
by josh
Fix the problem at it's roots. If the problem is caused by the browser, and the whole reason you used a browser originally was to work around another problem (an unfinished cron script)... why not go and fix the problem (rewrite the code in such a way that the work happens as part of a cron script with no manual intervention needed.) Your client wants a cron script that works. He doesn't want his business hinged on some brittle arcane browser time out setting so that this problem regresses when it needs to be run on someone else's machine, or when you aren't around.

If the "problem" that you face now (browser timeout) was a "solution" 5 minutes ago (to a cron script that is not modular enough to do it's own work)... well 5 minutes ago you were probably wrong.