Bulk Data Transfer - Web site too busy error

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
zahidrahim
Forum Newbie
Posts: 4
Joined: Sat Dec 05, 2009 7:17 am

Bulk Data Transfer - Web site too busy error

Post by zahidrahim »

Hi Php Gurrus,
On my local network i created a php script which gets record from SQL Server and post those in my Oracle Database. it runs very fine for thousands of records but when the records exceeds to millions of records it start giving error Server too busy etc. I tried by increasing the max execution time to 120 seconds but in vain.....

I have Win2003 with zend core for Oracle installed.
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: Bulk Data Transfer - Web site too busy error

Post by requinix »

You're overwhelming the server. Don't try to do everything at once.
zahidrahim
Forum Newbie
Posts: 4
Joined: Sat Dec 05, 2009 7:17 am

Re: Bulk Data Transfer - Web site too busy error

Post by zahidrahim »

If there is a requirement to do everything at once then what is the solution to transfer bulk data without error.
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: Bulk Data Transfer - Web site too busy error

Post by requinix »

By not doing everything at once I mean giving the servers a break every once in a while. Sure, you can fire off 100 commands a second, but if the server can only handle 99 a second with a buffer of 50 then 51 seconds in you'll get a Busy notice.
Just slow down. You can do 95 a second, or do 100 for 50 seconds and then back off and let it handle everything before doing the next 100.

If you want better servers, upgrade. More memory, better processors, faster hard drives... Beyond that you can't just make it go faster.
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: Bulk Data Transfer - Web site too busy error

Post by Eran »

If you can, offload any logic from PHP to MySQL. MySQL is a daemon, it does not have a time limit and it can process your requests for as long as it needs to (or until the server overloads). Consider optimizing your process to minimize the load
Post Reply