Read large tab delimited text file & write MYSQL database
Moderator: General Moderators
Read large tab delimited text file & write MYSQL database
We already have a table set up, and we have a large txt file that we are FTPing to our server. The file has over one million rows of data with 27 fields all tab delimited. I have never processed such a large file and was wondering the best method to buffer it so that it does not bog down my server. Could someone point me to a good tutorial or provide information on how to best do this?
Re: Read large tab delimited text file & write MYSQL databas
What have you tried? What errors have you encountered? My first approach would be something like mysqlimport or LOAD DATA
Re: Read large tab delimited text file & write MYSQL databas
fopen(), fgetcsv() should iterate through the file one line at a time.
Real programmers don't comment their code. If it was hard to write, it should be hard to understand.
- Christopher
- Site Administrator
- Posts: 13596
- Joined: Wed Aug 25, 2004 7:54 pm
- Location: New York, NY, US
Re: Read large tab delimited text file & write MYSQL databas
I hope you are kidding pickle! PHP looping thorough a million line file? I would also recommend mysqlimport or LOAD DATA INFILE.pickle wrote:fopen(), fgetcsv() should iterate through the file one line at a time.
(#10850)
Re: Read large tab delimited text file & write MYSQL databas
If he just needs to do a raw import - ya, that's a waste of resources. If computation needs to be done on the data though - there's not much choice.Christopher wrote:I hope you are kidding pickle! PHP looping thorough a million line file? I would also recommend mysqlimport or LOAD DATA INFILE.pickle wrote:fopen(), fgetcsv() should iterate through the file one line at a time.
Real programmers don't comment their code. If it was hard to write, it should be hard to understand.