CSV Import Problem to MySQL

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
will808
Forum Newbie
Posts: 8
Joined: Mon Apr 28, 2008 12:42 pm

CSV Import Problem to MySQL

Post by will808 »

Hi,

Apologies if this has been posted before - I've searched through the forum and googled, but as I don't exactly know what I'm looking for, the results haven't been great. As you'll gather, I'm totally new to PHP, so please bear with me...

I'm trying to import a .csv into MySQL using the PHP script below. The script is working fine except for the fact that some columns are spilling over into the next column during the import. This must be to do with the way that the csv is formatted (separated by ',' "enclosed by '"'). Just wondering if anyone could tell me what I need to edit in the script below to take account of this:

Code: Select all

 
<?
 
 
 
/********************************************************************************************/
 
/* Code at http://legend.ws/blog/tips-tricks/csv-php-mysql-import/
 
/* Edit the entries below to reflect the appropriate values
 
/********************************************************************************************/
 
$databasehost = "";
 
$databasename = "";
 
$databasetable = "";
 
$databaseusername ="";
 
$databasepassword = "";
 
$fieldseparator = ",";
 
$lineseparator = "\n";
 
$csvfile = "upload/export.csv";
 
/********************************************************************************************/
 
/* Would you like to add an ampty field at the beginning of these records?
 
/* This is useful if you have a table with the first field being an auto_increment integer
 
/* and the csv file does not have such as empty field before the records.
 
/* Set 1 for yes and 0 for no. ATTENTION: don't set to 1 if you are not sure.
 
/* This can dump data in the wrong fields if this extra field does not exist in the table
 
/********************************************************************************************/
 
$addauto = 0;
 
/********************************************************************************************/
 
/* Would you like to save the mysql queries in a file? If yes set $save to 1.
 
/* Permission on the file should be set to 777. Either upload a sample file through ftp and
 
/* change the permissions, or execute at the prompt: touch output.sql && chmod 777 output.sql
 
/********************************************************************************************/
 
$save = 1;
 
$outputfile = "output.sql";
 
/********************************************************************************************/
 
 
 
 
 
if(!file_exists($csvfile)) {
 
    echo "File not found. Make sure you specified the correct path.\n";
 
    exit;
 
}
 
 
 
$file = fopen($csvfile,"r");
 
 
 
if(!$file) {
 
    echo "Error opening data file.\n";
 
    exit;
 
}
 
 
 
$size = filesize($csvfile);
 
 
 
if(!$size) {
 
    echo "File is empty.\n";
 
    exit;
 
}
 
 
 
$csvcontent = fread($file,$size);
 
 
 
fclose($file);
 
 
 
$con = @mysql_connect($databasehost,$databaseusername,$databasepassword) or die(mysql_error());
 
@mysql_select_db($databasename) or die(mysql_error());
 
 
 
$lines = 0;
 
$queries = "";
 
$linearray = array();
 
 
 
foreach(split($lineseparator,$csvcontent) as $line) {
 
 
 
    $lines++;
 
 
 
    $line = trim($line," \t");
 
    
 
    $line = str_replace("\r","",$line);
 
    
 
    /************************************************************************************************************
 
    This line escapes the special character. remove it if entries are already escaped in the csv file
 
    ************************************************************************************************************/
 
    $line = str_replace("'","\'",$line);
 
    /***********************************************************************************************************/
 
    
 
    $linearray = explode($fieldseparator,$line);
 
    
 
    $linemysql = implode("','",$linearray);
 
    
 
    if($addauto)
 
        $query = "insert into $databasetable values('','$linemysql');";
 
    else
 
        $query = "insert into $databasetable values('$linemysql');";
 
    
 
    $queries .= $query . "\n";
 
 
 
    @mysql_query($query);
 
}
 
 
 
@mysql_close($con);
 
 
 
if($save) {
 
    
 
    if(!is_writable($outputfile)) {
 
        echo "File is not writable, check permissions.\n";
 
    }
 
    
 
    else {
 
        $file2 = fopen($outputfile,"w");
 
        
 
        if(!$file2) {
 
            echo "Error writing to the output file.\n";
 
        }
 
        else {
 
            fwrite($file2,$queries);
 
            fclose($file2);
 
        }
 
    }
 
    
 
}
 
 
 
echo "Found a total of $lines records in this csv file.\n";
 
 
 
 
 
?>
 

If it's of any use, the following command works fine if I run it from the db console. The fact that this works suggests to me that there is something missing in the script that deals with the formatting?

Code: Select all

 
load data local infile '/upload/export.csv' into table Test fields terminated by ',' enclosed by '"' lines terminated by '\n' ignore 1 lines (Field1, Field2, Field2, etc);
 
Thanks for any help.
User avatar
yacahuma
Forum Regular
Posts: 870
Joined: Sun Jul 01, 2007 7:11 am

Re: CSV Import Problem to MySQL

Post by yacahuma »

Is this a one time deal?
If it is just us navicat(comercial product with 30 day demo), wizard import
will808
Forum Newbie
Posts: 8
Joined: Mon Apr 28, 2008 12:42 pm

Re: CSV Import Problem to MySQL

Post by will808 »

Hi,

No it's something I'll want to do regularly.
User avatar
yacahuma
Forum Regular
Posts: 870
Joined: Sun Jul 01, 2007 7:11 am

Re: CSV Import Problem to MySQL

Post by yacahuma »

this you check your csv file so that it does not contain the character you are using to split the columns?
will808
Forum Newbie
Posts: 8
Joined: Mon Apr 28, 2008 12:42 pm

Re: CSV Import Problem to MySQL

Post by will808 »

Hi,

Yes, I think you are right - I think the CSV contains the separator where it should not be. The CSV is generated for me, so this is not something I can change.

But - because the "load data local infile" method works if I enter it in the console as in my post above, I'm thinking there must be a way to edit the script so that it works in the same way?
User avatar
RobertGonzalez
Site Administrator
Posts: 14293
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Re: CSV Import Problem to MySQL

Post by RobertGonzalez »

PHP has built in csv file handling functions. You might to try those instead of using regular file parsing against the separator.
will808
Forum Newbie
Posts: 8
Joined: Mon Apr 28, 2008 12:42 pm

Re: CSV Import Problem to MySQL

Post by will808 »

OK, seems the problem was quotes in the data, but I can't change that as the CSV is supplied to me. I've found this script which does the job perfectly. The problem is I don't know what I need to do to load this array into the database. I've only shown the function below - connecting to the DB fine and executing other queries on it, so could anyone help me with this:

Code: Select all

 
<?php
    function getCSVValues($string, $separator=",")
    {
        $elements = explode($separator, $string);
        
        for ($i = 0; $i < count($elements); $i++) 
        {
            $nquotes = substr_count($elements[$i], '"');
            
            if ($nquotes %2 == 1)
            {
                for ($j = $i+1; $j < count($elements); $j++) 
                {
                    if (substr_count($elements[$j], '"') > 0) 
                    {
                        // Put the quoted string's pieces back together again
                        array_splice($elements, $i, $j-$i+1,
                        implode($separator, array_slice($elements, $i, $j-$i+1)));
                        break;
                    }
                }
            }
            
            if ($nquotes > 0) 
            {
                // Remove first and last quotes, then merge pairs of quotes
                $qstr =& $elements[$i];
                $qstr = substr_replace($qstr, '', strpos($qstr, '"'), 1);
                $qstr = substr_replace($qstr, '', strrpos($qstr, '"'), 1);
                $qstr = str_replace('""', '"', $qstr);
            }
        }
        
        return $elements;
    }
    
    $file = file_get_contents('quotes.csv');
    $dataStrings = explode("\r", $file);
    
    $i = 0;
    foreach ( $dataStrings as $data ) ++$i; 
 
    for ( $j = 0; $j < $i; ++$j )
    {
        $strings = getCSVValues( $dataStrings[$j] );
        print_r($strings);
        print("<br/>\n\r"); [color=#FF0000]//this is where I want to insert into the db rather than print.[/color]
 
    }
?>
 
Thanks for any help.
User avatar
RobertGonzalez
Site Administrator
Posts: 14293
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Re: CSV Import Problem to MySQL

Post by RobertGonzalez »

fgetcsv()... same thing, only built in to PHP so it is way faster.

After you have each row do what you will with it.
User avatar
yacahuma
Forum Regular
Posts: 870
Joined: Sun Jul 01, 2007 7:11 am

Re: CSV Import Problem to MySQL

Post by yacahuma »

just pre-parse the data. replace the problematic character with something else and then use php built in functions.
matthewl
Forum Newbie
Posts: 13
Joined: Sat May 03, 2008 5:28 am

Re: CSV Import Problem to MySQL

Post by matthewl »

why not just have php upload the file and do a system call to load data in mysql as if it was at the commandline..

It will save you a lot of code and will probably run a lot faster than getting php to do so much work.
Post Reply