Page 1 of 1
Executing PHP Script without timeouts
Posted: Thu Mar 05, 2009 2:55 pm
by markwelch
I'm still very new to LAMP, and after many searches I cannot find any effective way to run a script that's going to take longer than the timeout limit. Specifically, I am trying to import a series of files, some of which are very large, into a database. I have a PHP script to "import one file" but sometimes it times out; and since I have hundreds of files to import, I'd like to automate the "launching" of the import process, either with a single script that loops through each file, or by launching a whole bunch of processes to import the files. (I've been told in the past that running a bunch of concurrent processes will result in faster import; this seems counter-intuitive to me, but I can't get to that issue until I can find a way to run processes that would exceed the timeout limit.)
Of course, when I just try to run the PHP script from a browser window, it times out (either after importing only a few small files, or while importing a single large file).
I do NOT believe that it would be appropriate to change the default time-out limit (currently 30 seconds). Some of the larger files may take many minutes, perhaps an hour or longer, to import.
I know that I'll need to revise my scripts so that all output goes to a log file instead of echoing to the browser screen; I'd like to find some way to track and manage the progress of any pending imports.
I keep finding references to the Linux "cron" command, but right now I am not trying to schedule the execution time, but simply to "do this now" so cron doesn't seem like the right tool for this job.
I've now spent dozens of hours trying to find a solution, including paying people for (bad) advice, but have not yet found a solution to what "seems like" a relatively common and simple requirement.
Thanks in advance for any guidance.
Re: Executing PHP Script without timeouts
Posted: Thu Mar 05, 2009 3:05 pm
by pickle
Did a Google search for "PHP timeout". The first hit was to
set_time_limit()
Re: Executing PHP Script without timeouts
Posted: Thu Mar 05, 2009 3:10 pm
by markwelch
Thanks for the quick reply. Yes, I'm aware that there is a "set_time_limit" function, but it is constrained by the "max_execution_time" setting -- which in turn, is ignored if the script is run "from the command line." I haven't the slightest idea how to run a PHP script from the command line (I assume that would require me to putty to the server and execute commands manually -- not very helpful when I'm trying to automate a process).
So basically, I could set a time limit such as "9999" in my script, but it would still timeout when the "max_execution_time" is reached.
I've been struggling with this problem now for three weeks, and I just keep getting frustrated and moving to some other project. Over and over, I am told that this is a simple problem -- but even when I have agreed to PAY someone for the solution, they just flake out.
Re: Executing PHP Script without timeouts
Posted: Thu Mar 05, 2009 3:19 pm
by pickle
Why not run this from the browser? If you're not automating this process, the browser's as good a place to start the process as any.
You can set the timeout to 0 which, if memory serves me correctly, lets it run forever - even beyond the max_execution_time limit.
I just had a thought. Sadly it's easier to describe in pseudo-code than in words:
File accessed through browser
Code: Select all
<?php
$files_to_import = glob('some/directory/*');
foreach($files_to_import as $filename)
{
shell_exec('php5 file_that_imports_one_file.php -'.$filename);
}
?>
File that imports one file
Code: Select all
<?php
$filename = $argv[1];
...
?>
This way, it may take more than max_execution_time for the page in the browser to appear, but pretty much all of that wait will be for a shell command to finish - which doesn't affect the execution time of the original PHP script.
Re: Executing PHP Script without timeouts
Posted: Thu Mar 05, 2009 3:35 pm
by markwelch
So I need to re-code the current "import one file" function so that it is a stand-alone script (not a function), and collect the filename from $argv[1] instead of from a function parameter. I'll give it a try.
Added: Hmm, shell_exec is disabled. Now I need to find where php.ini is (again).
Re: Executing PHP Script without timeouts
Posted: Thu Mar 05, 2009 4:03 pm
by pickle
Make a page that shows phpinfo() - that'll tell you what ini file to modify.
Re: Executing PHP Script without timeouts
Posted: Thu Mar 05, 2009 4:20 pm
by markwelch
Found php.ini, changed it.
Scripts "seem" to run normally, to completion, but nothing is getting stored into the database. Now I need to re-code the entire script to log to a text file so I can see what (if anything) is getting executed.
Re: Executing PHP Script without timeouts
Posted: Thu Mar 05, 2009 4:38 pm
by markwelch
Hmm, now I've re-written the script to open a logfile and write to it -- but no logfiles are created or written. It's as if the shell_exec isn't getting executed at all (but the error messages that appeared earlier, noting that shell_exec was disabled, are no longer displayed).
Any suggestions on how to debug a script that's running under shell_exec? Is there any way to even tell if the script is getting run at all?
Re: Executing PHP Script without timeouts
Posted: Thu Mar 05, 2009 4:49 pm
by markwelch
I suspect the problem may be that I simply accepted the suggestion that a shell_exec parameter is accessible within a PHP script as $argv[1] -- I cannot find any documentation to this effect.
In the "import one file" script, I have this code to determine the correct file name (for merchant number 5008 the file would be at /home/webadmin/feeds-sas/txt/5008.txt).
$datafeed_file_name = $argv[1] . ".txt" ;
$datafeed_file_path = "/home/webadmin/feeds-sas/txt/" ;
$datafeed_file = $datafeed_file_path . $datafeed_file_name ;
but of course $argv[1] has never been defined anywhere.
I am calling the "import one file" script using this PHP code:
shell_exec ('php import_sas_datafeed.php -5008') ;
I don't really know if either of these strategies is valid. Maybe the dash before the number should not be there? Maybe the number should be in quotes?
Certainly, no log files are being written.