Page 1 of 1

Splitting a file after a new line

Posted: Wed Mar 26, 2008 2:33 pm
by tarja311
Hey guys,

I have a script that reads in data from a flat text file. What it does is it chops the file up into chunks based on the file-size that it is given. For instance, if i set the file-size limit to 80k, it is going to split the file when it reaches 80k, and will continue to do so until it reaches the end of the flat text file.

It looks something like this:
- "rawtextfile.txt" (goes into script ...)

and what comes out of it looks like this:
- "rawtextfile.txt.chunk1" (80k)
- "rawtextfile.txt.chunk2" (80k)
- "rawtextfile.txt.chunk3" (whatever bytes left over ...)


What i want it to do is i want it to split the file as soon as it reaches the end of the line rather than splitting it exactly at 80k. I ask this because it is chopping right through a line and continuing that line in the next file. I want it to reach the end of the line first before splitting the file.

Is that possible?

Here is what i have so far:

Code: Select all

 
function file_chop_big($file_path, $chunk_size) {
  $size = filesize($file_path);
 
  //find number of full $chunk_size byte portions
  $num_chunks = floor($size / $chunk_size);
 
  $file_handle = fopen($file_path, 'r'); 
 
  $chunks = array();
 
  for($kk = 0; $kk < $num_chunks; $kk++) {
    $chunks[$kk] = basename($file_path).'.chunk'.($kk + 1);
    $chunk_handle = fopen($chunks[$kk], 'w');   //open the chunk file for writing
 
    //write the data to the chunk file 1k at a time
    while((ftell($chunk_handle) + 1024) <= $chunk_size) {
      fwrite($chunk_handle, fread($file_handle, 1024));
    } // end while-loop.
 
    if(($leftover = $chunk_size - ftell($chunk_handle)) > 0 ) {
      fwrite($chunk_handle, fread($file_handle, $leftover));
    } // end if.
 
    fclose($chunk_handle);
  } // end for-loop.
 
  if(($leftover = $size - ftell($file_handle)) > 0) {
    $chunks[$num_chunks] = basename($file_path).'.chunk'.($num_chunks + 1);
    $chunk_handle = fopen($chunks[$num_chunks], 'w');
    
    while(!feof($file_handle)) {
      fwrite($chunk_handle, fread($file_handle, 1024));
    } // end while-loop.
 
    fclose($chunk_handle);
  } // end if.
 
  fclose($file_handle);
 
  return $chunks;
} // end function.
 
Any ideas?

Thanks

Re: Splitting a file after a new line

Posted: Wed Mar 26, 2008 4:24 pm
by pickle
Write 80KB to the file, then keep writing until you hit a newline character. You'll probably have to change your for loop to a while loop to get it done.

Re: Splitting a file after a new line

Posted: Wed Mar 26, 2008 4:39 pm
by tarja311
Hmm. Could you please show me an example in code?

Re: Splitting a file after a new line

Posted: Wed Mar 26, 2008 6:13 pm
by pickle
Nope - that's for you to figure out (plus I don't have any code) - I'm just here to give you suggestions.

Re: Splitting a file after a new line

Posted: Wed Mar 26, 2008 9:27 pm
by tarja311
Darn, i was afraid of that. I can't believe for something as simple as this i am just so completely stumped on. I've been screwing around with this script all day. Oy! :(

Re: Splitting a file after a new line

Posted: Thu Mar 27, 2008 9:49 am
by pickle
Whenever I get stumped I stop trying to figure out the problem exactly, and plan out the general idea. It's a bit like writing an outline for an essay before writing the actual essay - it helps you get the layout of the problem squared away without getting distracted by the details.