Read large tab delimited text file & write MYSQL database

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
cebuy
Forum Newbie
Posts: 1
Joined: Sun Feb 09, 2014 1:46 am

Read large tab delimited text file & write MYSQL database

Post by cebuy »

We already have a table set up, and we have a large txt file that we are FTPing to our server. The file has over one million rows of data with 27 fields all tab delimited. I have never processed such a large file and was wondering the best method to buffer it so that it does not bog down my server. Could someone point me to a good tutorial or provide information on how to best do this?
User avatar
Celauran
Moderator
Posts: 6427
Joined: Tue Nov 09, 2010 2:39 pm
Location: Montreal, Canada

Re: Read large tab delimited text file & write MYSQL databas

Post by Celauran »

What have you tried? What errors have you encountered? My first approach would be something like mysqlimport or LOAD DATA
User avatar
pickle
Briney Mod
Posts: 6445
Joined: Mon Jan 19, 2004 6:11 pm
Location: 53.01N x 112.48W
Contact:

Re: Read large tab delimited text file & write MYSQL databas

Post by pickle »

fopen(), fgetcsv() should iterate through the file one line at a time.
Real programmers don't comment their code. If it was hard to write, it should be hard to understand.
User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Re: Read large tab delimited text file & write MYSQL databas

Post by Christopher »

pickle wrote:fopen(), fgetcsv() should iterate through the file one line at a time.
I hope you are kidding pickle! PHP looping thorough a million line file? I would also recommend mysqlimport or LOAD DATA INFILE.
(#10850)
User avatar
pickle
Briney Mod
Posts: 6445
Joined: Mon Jan 19, 2004 6:11 pm
Location: 53.01N x 112.48W
Contact:

Re: Read large tab delimited text file & write MYSQL databas

Post by pickle »

Christopher wrote:
pickle wrote:fopen(), fgetcsv() should iterate through the file one line at a time.
I hope you are kidding pickle! PHP looping thorough a million line file? I would also recommend mysqlimport or LOAD DATA INFILE.
If he just needs to do a raw import - ya, that's a waste of resources. If computation needs to be done on the data though - there's not much choice.
Real programmers don't comment their code. If it was hard to write, it should be hard to understand.
Post Reply