Page 1 of 1

Bulk Data Transfer - Web site too busy error

Posted: Tue Dec 29, 2009 5:23 am
by zahidrahim
Hi Php Gurrus,
On my local network i created a php script which gets record from SQL Server and post those in my Oracle Database. it runs very fine for thousands of records but when the records exceeds to millions of records it start giving error Server too busy etc. I tried by increasing the max execution time to 120 seconds but in vain.....

I have Win2003 with zend core for Oracle installed.

Re: Bulk Data Transfer - Web site too busy error

Posted: Tue Dec 29, 2009 5:25 am
by requinix
You're overwhelming the server. Don't try to do everything at once.

Re: Bulk Data Transfer - Web site too busy error

Posted: Tue Dec 29, 2009 11:27 pm
by zahidrahim
If there is a requirement to do everything at once then what is the solution to transfer bulk data without error.

Re: Bulk Data Transfer - Web site too busy error

Posted: Wed Dec 30, 2009 3:25 am
by requinix
By not doing everything at once I mean giving the servers a break every once in a while. Sure, you can fire off 100 commands a second, but if the server can only handle 99 a second with a buffer of 50 then 51 seconds in you'll get a Busy notice.
Just slow down. You can do 95 a second, or do 100 for 50 seconds and then back off and let it handle everything before doing the next 100.

If you want better servers, upgrade. More memory, better processors, faster hard drives... Beyond that you can't just make it go faster.

Re: Bulk Data Transfer - Web site too busy error

Posted: Wed Dec 30, 2009 3:28 am
by Eran
If you can, offload any logic from PHP to MySQL. MySQL is a daemon, it does not have a time limit and it can process your requests for as long as it needs to (or until the server overloads). Consider optimizing your process to minimize the load