Page 1 of 1
Read large tab delimited text file & write MYSQL database
Posted: Sun Feb 09, 2014 2:43 am
by cebuy
We already have a table set up, and we have a large txt file that we are FTPing to our server. The file has over one million rows of data with 27 fields all tab delimited. I have never processed such a large file and was wondering the best method to buffer it so that it does not bog down my server. Could someone point me to a good tutorial or provide information on how to best do this?
Re: Read large tab delimited text file & write MYSQL databas
Posted: Sun Feb 09, 2014 6:58 am
by Celauran
What have you tried? What errors have you encountered? My first approach would be something like mysqlimport or
LOAD DATA
Re: Read large tab delimited text file & write MYSQL databas
Posted: Mon Feb 10, 2014 1:46 pm
by pickle
fopen(), fgetcsv() should iterate through the file one line at a time.
Re: Read large tab delimited text file & write MYSQL databas
Posted: Tue Feb 11, 2014 9:39 pm
by Christopher
pickle wrote:fopen(), fgetcsv() should iterate through the file one line at a time.
I hope you are kidding pickle! PHP looping thorough a million line file? I would also recommend mysqlimport or LOAD DATA INFILE.
Re: Read large tab delimited text file & write MYSQL databas
Posted: Wed Feb 12, 2014 10:02 am
by pickle
Christopher wrote:pickle wrote:fopen(), fgetcsv() should iterate through the file one line at a time.
I hope you are kidding pickle! PHP looping thorough a million line file? I would also recommend mysqlimport or LOAD DATA INFILE.
If he just needs to do a raw import - ya, that's a waste of resources. If computation needs to be done on the data though - there's not much choice.