Page 1 of 1

Read large tab delimited text file & write MYSQL database

Posted: Sun Feb 09, 2014 2:43 am
by cebuy
We already have a table set up, and we have a large txt file that we are FTPing to our server. The file has over one million rows of data with 27 fields all tab delimited. I have never processed such a large file and was wondering the best method to buffer it so that it does not bog down my server. Could someone point me to a good tutorial or provide information on how to best do this?

Re: Read large tab delimited text file & write MYSQL databas

Posted: Sun Feb 09, 2014 6:58 am
by Celauran
What have you tried? What errors have you encountered? My first approach would be something like mysqlimport or LOAD DATA

Re: Read large tab delimited text file & write MYSQL databas

Posted: Mon Feb 10, 2014 1:46 pm
by pickle
fopen(), fgetcsv() should iterate through the file one line at a time.

Re: Read large tab delimited text file & write MYSQL databas

Posted: Tue Feb 11, 2014 9:39 pm
by Christopher
pickle wrote:fopen(), fgetcsv() should iterate through the file one line at a time.
I hope you are kidding pickle! PHP looping thorough a million line file? I would also recommend mysqlimport or LOAD DATA INFILE.

Re: Read large tab delimited text file & write MYSQL databas

Posted: Wed Feb 12, 2014 10:02 am
by pickle
Christopher wrote:
pickle wrote:fopen(), fgetcsv() should iterate through the file one line at a time.
I hope you are kidding pickle! PHP looping thorough a million line file? I would also recommend mysqlimport or LOAD DATA INFILE.
If he just needs to do a raw import - ya, that's a waste of resources. If computation needs to be done on the data though - there's not much choice.