PHP + MySQL + Large Text File = Unknown result.

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: PHP + MySQL + Large Text File = Unknown result.

Post by Eran »

probably running out of memory. try to add a stopping condition - put in a small counter, and break the loop after a number iterations. Try to figure out when exactly it craps out. also run memory_get_usage() just after the loops ends and before the script exists.

You should probably increase php memory allocation for this operation, the default is very low
Benwahballz
Forum Commoner
Posts: 25
Joined: Mon Sep 21, 2009 12:54 pm

Re: PHP + MySQL + Large Text File = Unknown result.

Post by Benwahballz »

I put a counter inside the while loop. Here are some benchmark results ( I ran each test twice for each number of lines)

Code: Select all

 
Memory Used 198536 on 100000 Lines
 

Code: Select all

 
Memory Used 198256 on 100000 Lines
 

Code: Select all

 
Memory Used 289844 on 200000 Lines
 

Code: Select all

 
Memory Used 289844 on 200000 Lines
 
I tried running the code at 300,000 Lines, but 4 out of 5 times it died with no result. The ONE time it did work I got:

Code: Select all

 
Memory Used 367772 on 300000 Lines
 
Toned back to 250,000 Lines

Code: Select all

 
Memory Used 342176 on 250000 Lines
 

Code: Select all

 
Memory Used 342176 on 250000 Lines
 
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: PHP + MySQL + Large Text File = Unknown result.

Post by Eran »

Very weird indeed :)

Are you running php with display_errors = on? also check your apache error logs after running the loop (and it fails) to see if there's anything wrong there.
Benwahballz
Forum Commoner
Posts: 25
Joined: Mon Sep 21, 2009 12:54 pm

Re: PHP + MySQL + Large Text File = Unknown result.

Post by Benwahballz »

FYI:

What I ended up doing was splitting up the files into appx 7-8MB each, which works out to about 40 files.

What I have it doing is checking if the file exists and pushing the data into the database. Once the file is complete, I go on to the next one if there is a file that exists. I have a little piece of java code that once it completes the file it automatically forwards to the next page with the file number of the next file to load data from.

Every fifth file I have it sleep for 10 seconds and continue on.

Couldnt really think of any other workaround, and this works 95% of the time at least. Every now and then it will stop every 15 files or so, but I can just refresh the page and it will continue on from there.



If anyone else has any other suggestions or comments it would be appreciated.

Thanks
Post Reply