Page 1 of 1

Loading text from an uploaded text file.

Posted: Fri Nov 25, 2011 12:26 pm
by rellison
Forgive if this has been asked I searched but could not find what i was looking for.

I have a database routine where a user will upload a text file that contains data. Each record of data in the text file ends with <EOR>
Each record can contain a random length of text possibly up to 3000 characters. And can have multiple CR's located in it.
I need to parse out the record up to and including the <EOR>. So once the file is read in up to the end of first <EOR> that string of data would be sent to the parse routines. Then the next record gets read in and so on till the end of the file.

I am using this a to do the routine now but the encountered CR's keep breaking the string to soon and not all of the record is getting read in...

Code: Select all

    		// Read the first line
    		$string = fgets ($file, 1024);
             	$string = strtoupper ($string);
                if (stristr($string, "<EOR>"))
        		{
        			// Process ADIF file
                               processFile ($file,$count,$duplicate,$Bad,$string,$usertable);
        		}
        		else
        		{
        			while (($string = fgets ($file,1024)) && !$valid_file)
        			{
        				$string = strtoupper ($string);
                                        if (stristr($string, "<EOR>") )
        			         {
					   $valid_file = 1;
                                           processFile ($file,$count,$duplicate,$Bad,$string,$usertable);
        				}
        			}
        
        			// No records in file found - exit with an error
        			if (!$valid_file)
        			{
        				echo "<P>Error - Unable to upload file: $browser_name\n";
        				echo "<P>Invalid file format.\n";
         				die();
        			}
        		}

Can someone show me the correct routine I should be using to do this.. This is driving me nuts since I am not that good with PHP yet..

Thanks Rick

Re: Loading text from an uploaded text file.

Posted: Fri Nov 25, 2011 12:48 pm
by Celauran
rellison wrote:the encountered CR's keep breaking the string to soon and not all of the record is getting read in...
fgets():
Reading ends when length - 1 bytes have been read, on a newline (which is included in the return value), or on EOF (whichever comes first). If no length is specified, it will keep reading from the stream until it reaches the end of the line.
Why not use file_get_contents() and explode()?

Re: Loading text from an uploaded text file.

Posted: Fri Nov 25, 2011 1:18 pm
by rellison
That might work but my worry is some of the text files being uploaded might contain 100,000records or more. and the memory requirements on the server..

Re: Loading text from an uploaded text file.

Posted: Sat Dec 10, 2011 5:35 pm
by rellison
I just wanted to Bump this as I have still not figured a good way to do this.Reading in the entire file and then exploding the contents really would not work with the large files that get uploaded. MY thinking is It should read in up to the <EOR> and include that into the string that is read in I just do not know how to code that..

TIA
Rick

Re: Loading text from an uploaded text file.

Posted: Sun Dec 11, 2011 2:50 pm
by McInfo
This is a generic enough problem that I decided to share my solution. The example displays its own source code.

Code: Select all

<?php
function main () {
    header('Content-Type: text/plain; charset=UTF-8');
    $file = fopen(basename(__FILE__), 'rb');
    chunkFile($file, 'echoRecord');
    fclose($file);
}
function echoRecord ($record) {
    echo $record, PHP_EOL;
}
function chunkFile ($handle, $callback, $delimiter=PHP_EOL, $chunkSize=1024) {
    $delimSize = strlen($delimiter);
    $lapSize = ($delimSize - 1) ?: 1;
    $chunk = '';
    $pos = 0;
    $record = '';
    while (!feof($handle)) {
        $chunk .= fread($handle, $chunkSize);
        $pos = strpos($chunk, $delimiter);
        if ($pos === false) {
            $record .= substr($chunk, 0, -$lapSize);
            $chunk = substr($chunk, -$lapSize);
        } else {
            $record .= substr($chunk, 0, $pos);
            $callback($record);
            $chunk = substr($chunk, $pos + $delimSize);
            $record = '';
        }
    }
    while ($chunk !== false) {
        $pos = strpos($chunk, $delimiter);
        if ($pos === false) {
            $callback($chunk);
            $chunk = false;
        } else {
            $callback(substr($chunk, 0, $pos));
            $chunk = substr($chunk, $pos + $delimSize);
        }
    }
}
main();
Let me know if you want it commented or if you find a problem.