50 Threads - all die - why?

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
MAtkins
Forum Newbie
Posts: 17
Joined: Fri Apr 02, 2010 3:55 pm

50 Threads - all die - why?

Post by MAtkins »

Hi:
OS is Windows Server 2003
I created an application that uses command line to start up 50 threads.
Each thread goes out to various Websites using cURL, retrieves the HTML.
It gets the records containing the URLs from MySql and enters results into MySql.

Each thread is called from a Web based PHP script.
Each thread is called asynchronously and the Web based script completes in about 1/2 minute.
I've verified that all 50 threads continue to run after the Web based script completes.

If I run it on about 2K of Websites it works fine, in less than 15 minutes.
WhenI bump it up to 65K of Websites all 50 threads quit, all at the same exact time, after about 1/2 hour.

At the top of the script used for each thread I have:
set_time_limit(0);

in php.ini I've got:
max_execution_time = 36000

I've got try catch blocks on all code that dumps errors into the database.
No catchable errors are thrown.
The CPU is dual core and it maxes out during the run from time to time for short periods.
The Page File goes up to about 1.6 gig for most of the run.
The hard drive still has about 50 gig to spare.

Again, all 50 threads quit at exactly the same time after about 1/2 hour.
What is making php quit?
Is there anything I can do to make it continue until the threads have completed?
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: 50 Threads - all die - why?

Post by requinix »

Set error_reporting to -1 and display_errors on. Then have the initial script capture each working script's output somewhere (eg, output0.txt, output1.txt, ...). Include stderr redirection, like

Code: Select all

exec("... >output$i.txt 2>&1");
Then check if there's anything in the files.
MAtkins
Forum Newbie
Posts: 17
Joined: Fri Apr 02, 2010 3:55 pm

Re: 50 Threads - all die - why?

Post by MAtkins »

Thanks for the reply.
This gives me some ideas but I've got a problem.

The initial script is Web based. I fires off the threads asynchronously so getting them to return anything isn't an option.

I've found a few errors like 'unidentified constant' where I neglected to add the $ to the beginning of a variable.
Fixing those has definitely made a difference so I"m getting that an error like that in one thread will kill all the threads.

Is there any way I can get each thread to write the errors to a log file?
Again, I am capturing errors that can be caught and saving them to a database.
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: 50 Threads - all die - why?

Post by requinix »

MAtkins wrote:Is there any way I can get each thread to write the errors to a log file?
Again, I am capturing errors that can be caught and saving them to a database.
Yeah... By modifying the initial script to use output redirection. Like I showed in my previous post. The exact mechanism depends on how you're starting each "thread".
MAtkins
Forum Newbie
Posts: 17
Joined: Fri Apr 02, 2010 3:55 pm

Re: 50 Threads - all die - why?

Post by MAtkins »

It's Windows. I'm not using exec.
I'm using this:
$WshShell = new COM("WScript.Shell");
$oExec = $WshShell->Run("php " . $base_path . "\\BacklinkSourceExtractor.php --JobID $JobID --JobThreadID $JobThreadID", 7, false);
MAtkins
Forum Newbie
Posts: 17
Joined: Fri Apr 02, 2010 3:55 pm

Re: 50 Threads - all die - why?

Post by MAtkins »

It is reporting to an error log but for this last run it didn't log anything.
The threads ran longer this time but I'm seeing no errors of any kind, still the threads stopped before completion.
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: 50 Threads - all die - why?

Post by requinix »

Okay, no exec, but really what you're doing is practically the same thing.

Code: Select all

$oExec = $WshShell->Run("php " . $base_path . "\\BacklinkSourceExtractor.php --JobID $JobID --JobThreadID $JobThreadID 1>output$JobID.txt 2>&1", 7, false);
MAtkins
Forum Newbie
Posts: 17
Joined: Fri Apr 02, 2010 3:55 pm

Re: 50 Threads - all die - why?

Post by MAtkins »

Thanks for sticking with me here. I wasn't sure if I could even find anybody who knew about threading.

Boy, I trashed my code and couldn't get it to work.
It's hard to get it to return compile time errors.

I finally did get it to run.
It looks like your solution made it stop working asynchronously though.
Now the Web script is holding on until all the threads have completed (this is with 2K records, not 65K).

Actually, all the threads have completed and the Web script is *still* running.
I just ran it again without the new output params and the Web script stopped correctly.
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: 50 Threads - all die - why?

Post by requinix »

Then there's a nuance I don't know of with WScript. Oh well.

Have you tried just running one thread on your own? Manually? Like from the command line?
Another idea: open your php.ini, change the error_log setting to "syslog", and trying stuff again. Hopefully it'll dump problems into Windows' event log (which you can see with the Event Viewer - run "eventvwr.msc" and check the Application section).
MAtkins
Forum Newbie
Posts: 17
Joined: Fri Apr 02, 2010 3:55 pm

Re: 50 Threads - all die - why?

Post by MAtkins »

Yea, running 1 thread at a time was how I found the problems I found so far.
I'd hate to have to run 50 threads like that but it may come to that.

I just ran the 65K again and it did it again, No errors of any kind that I can find but none of the threads completed.

I'll try to redirect the errors to the syslog and see if that helps.
MAtkins
Forum Newbie
Posts: 17
Joined: Fri Apr 02, 2010 3:55 pm

Re: 50 Threads - all die - why?

Post by MAtkins »

I redirected the errors to the syslog and that helped a great deal.
About 2/3 of the threads actually completed this time and I got some errors to work with.
The most common is that the database connection fails due to timeouts.

This is even though the database is on the same server as the php threads.
I'm not quite sure how to solve that but at least I know what the problem is.
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: 50 Threads - all die - why?

Post by requinix »

Do you need the database for the entire duration of the script? Take a look at mysql_ping.

Any way you can restructure the code to use the database for only a (relatively) small portion of time? It'd be a good idea, unless doing so would push your memory usage noticeably higher. For example, if you do a lot of INSERTs, push them all into a temporary file until the very end, then reload it and execute everything at once.
MAtkins
Forum Newbie
Posts: 17
Joined: Fri Apr 02, 2010 3:55 pm

Re: 50 Threads - all die - why?

Post by MAtkins »

Well, Windows pools connections. Generally when I connect to a database I always do it short term.
I open the connection, do the work and then close it.
I am putting a lot of inserts into it in the beginning but that's before I create the threads.
I drop 2K at a time in one sql statement.

The problem is when the thread reads those recs.
So, I insert 65K of records then allocate them to 50 threads 65K/50 then create the threads.
When each thread opens a connection to retrieve its allocated records that's when the trouble starts.
50 threads try to connect more or less simultaneously.
I've added a 2 second delay between each thread creation and that helped some.
I also added a loop so if the connection fails it'll try again 4 times before 'failing' and wait 2 seconds between each try.
That seems to help.

I'm dealing mostly now with memory allocation. It turns out that some of the sites have huge Web pages (mostly blog comments).
I'm wondering now about garbage collection & unset. Also, if memory allocation in php.ini is refers to all threads or each individual thread.
Post Reply