probably running out of memory. try to add a stopping condition - put in a small counter, and break the loop after a number iterations. Try to figure out when exactly it craps out. also run memory_get_usage() just after the loops ends and before the script exists.
You should probably increase php memory allocation for this operation, the default is very low
PHP + MySQL + Large Text File = Unknown result.
Moderator: General Moderators
-
Benwahballz
- Forum Commoner
- Posts: 25
- Joined: Mon Sep 21, 2009 12:54 pm
Re: PHP + MySQL + Large Text File = Unknown result.
I put a counter inside the while loop. Here are some benchmark results ( I ran each test twice for each number of lines)
I tried running the code at 300,000 Lines, but 4 out of 5 times it died with no result. The ONE time it did work I got:
Toned back to 250,000 Lines
Code: Select all
Memory Used 198536 on 100000 Lines
Code: Select all
Memory Used 198256 on 100000 Lines
Code: Select all
Memory Used 289844 on 200000 Lines
Code: Select all
Memory Used 289844 on 200000 Lines
Code: Select all
Memory Used 367772 on 300000 Lines
Code: Select all
Memory Used 342176 on 250000 Lines
Code: Select all
Memory Used 342176 on 250000 Lines
Re: PHP + MySQL + Large Text File = Unknown result.
Very weird indeed 
Are you running php with display_errors = on? also check your apache error logs after running the loop (and it fails) to see if there's anything wrong there.
Are you running php with display_errors = on? also check your apache error logs after running the loop (and it fails) to see if there's anything wrong there.
-
Benwahballz
- Forum Commoner
- Posts: 25
- Joined: Mon Sep 21, 2009 12:54 pm
Re: PHP + MySQL + Large Text File = Unknown result.
FYI:
What I ended up doing was splitting up the files into appx 7-8MB each, which works out to about 40 files.
What I have it doing is checking if the file exists and pushing the data into the database. Once the file is complete, I go on to the next one if there is a file that exists. I have a little piece of java code that once it completes the file it automatically forwards to the next page with the file number of the next file to load data from.
Every fifth file I have it sleep for 10 seconds and continue on.
Couldnt really think of any other workaround, and this works 95% of the time at least. Every now and then it will stop every 15 files or so, but I can just refresh the page and it will continue on from there.
If anyone else has any other suggestions or comments it would be appreciated.
Thanks
What I ended up doing was splitting up the files into appx 7-8MB each, which works out to about 40 files.
What I have it doing is checking if the file exists and pushing the data into the database. Once the file is complete, I go on to the next one if there is a file that exists. I have a little piece of java code that once it completes the file it automatically forwards to the next page with the file number of the next file to load data from.
Every fifth file I have it sleep for 10 seconds and continue on.
Couldnt really think of any other workaround, and this works 95% of the time at least. Every now and then it will stop every 15 files or so, but I can just refresh the page and it will continue on from there.
If anyone else has any other suggestions or comments it would be appreciated.
Thanks