Page 3 of 3

Re: PHP + MySQL + Large Text File = Unknown result.

Posted: Tue Sep 22, 2009 5:37 pm
by Eran
probably running out of memory. try to add a stopping condition - put in a small counter, and break the loop after a number iterations. Try to figure out when exactly it craps out. also run memory_get_usage() just after the loops ends and before the script exists.

You should probably increase php memory allocation for this operation, the default is very low

Re: PHP + MySQL + Large Text File = Unknown result.

Posted: Tue Sep 22, 2009 6:09 pm
by Benwahballz
I put a counter inside the while loop. Here are some benchmark results ( I ran each test twice for each number of lines)

Code: Select all

 
Memory Used 198536 on 100000 Lines
 

Code: Select all

 
Memory Used 198256 on 100000 Lines
 

Code: Select all

 
Memory Used 289844 on 200000 Lines
 

Code: Select all

 
Memory Used 289844 on 200000 Lines
 
I tried running the code at 300,000 Lines, but 4 out of 5 times it died with no result. The ONE time it did work I got:

Code: Select all

 
Memory Used 367772 on 300000 Lines
 
Toned back to 250,000 Lines

Code: Select all

 
Memory Used 342176 on 250000 Lines
 

Code: Select all

 
Memory Used 342176 on 250000 Lines
 

Re: PHP + MySQL + Large Text File = Unknown result.

Posted: Tue Sep 22, 2009 6:42 pm
by Eran
Very weird indeed :)

Are you running php with display_errors = on? also check your apache error logs after running the loop (and it fails) to see if there's anything wrong there.

Re: PHP + MySQL + Large Text File = Unknown result.

Posted: Thu Sep 24, 2009 5:59 pm
by Benwahballz
FYI:

What I ended up doing was splitting up the files into appx 7-8MB each, which works out to about 40 files.

What I have it doing is checking if the file exists and pushing the data into the database. Once the file is complete, I go on to the next one if there is a file that exists. I have a little piece of java code that once it completes the file it automatically forwards to the next page with the file number of the next file to load data from.

Every fifth file I have it sleep for 10 seconds and continue on.

Couldnt really think of any other workaround, and this works 95% of the time at least. Every now and then it will stop every 15 files or so, but I can just refresh the page and it will continue on from there.



If anyone else has any other suggestions or comments it would be appreciated.

Thanks