Splitting a file after a new line

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
tarja311
Forum Commoner
Posts: 73
Joined: Fri Oct 20, 2006 10:57 pm

Splitting a file after a new line

Post by tarja311 »

Hey guys,

I have a script that reads in data from a flat text file. What it does is it chops the file up into chunks based on the file-size that it is given. For instance, if i set the file-size limit to 80k, it is going to split the file when it reaches 80k, and will continue to do so until it reaches the end of the flat text file.

It looks something like this:
- "rawtextfile.txt" (goes into script ...)

and what comes out of it looks like this:
- "rawtextfile.txt.chunk1" (80k)
- "rawtextfile.txt.chunk2" (80k)
- "rawtextfile.txt.chunk3" (whatever bytes left over ...)


What i want it to do is i want it to split the file as soon as it reaches the end of the line rather than splitting it exactly at 80k. I ask this because it is chopping right through a line and continuing that line in the next file. I want it to reach the end of the line first before splitting the file.

Is that possible?

Here is what i have so far:

Code: Select all

 
function file_chop_big($file_path, $chunk_size) {
  $size = filesize($file_path);
 
  //find number of full $chunk_size byte portions
  $num_chunks = floor($size / $chunk_size);
 
  $file_handle = fopen($file_path, 'r'); 
 
  $chunks = array();
 
  for($kk = 0; $kk < $num_chunks; $kk++) {
    $chunks[$kk] = basename($file_path).'.chunk'.($kk + 1);
    $chunk_handle = fopen($chunks[$kk], 'w');   //open the chunk file for writing
 
    //write the data to the chunk file 1k at a time
    while((ftell($chunk_handle) + 1024) <= $chunk_size) {
      fwrite($chunk_handle, fread($file_handle, 1024));
    } // end while-loop.
 
    if(($leftover = $chunk_size - ftell($chunk_handle)) > 0 ) {
      fwrite($chunk_handle, fread($file_handle, $leftover));
    } // end if.
 
    fclose($chunk_handle);
  } // end for-loop.
 
  if(($leftover = $size - ftell($file_handle)) > 0) {
    $chunks[$num_chunks] = basename($file_path).'.chunk'.($num_chunks + 1);
    $chunk_handle = fopen($chunks[$num_chunks], 'w');
    
    while(!feof($file_handle)) {
      fwrite($chunk_handle, fread($file_handle, 1024));
    } // end while-loop.
 
    fclose($chunk_handle);
  } // end if.
 
  fclose($file_handle);
 
  return $chunks;
} // end function.
 
Any ideas?

Thanks
User avatar
pickle
Briney Mod
Posts: 6445
Joined: Mon Jan 19, 2004 6:11 pm
Location: 53.01N x 112.48W
Contact:

Re: Splitting a file after a new line

Post by pickle »

Write 80KB to the file, then keep writing until you hit a newline character. You'll probably have to change your for loop to a while loop to get it done.
Real programmers don't comment their code. If it was hard to write, it should be hard to understand.
tarja311
Forum Commoner
Posts: 73
Joined: Fri Oct 20, 2006 10:57 pm

Re: Splitting a file after a new line

Post by tarja311 »

Hmm. Could you please show me an example in code?
User avatar
pickle
Briney Mod
Posts: 6445
Joined: Mon Jan 19, 2004 6:11 pm
Location: 53.01N x 112.48W
Contact:

Re: Splitting a file after a new line

Post by pickle »

Nope - that's for you to figure out (plus I don't have any code) - I'm just here to give you suggestions.
Real programmers don't comment their code. If it was hard to write, it should be hard to understand.
tarja311
Forum Commoner
Posts: 73
Joined: Fri Oct 20, 2006 10:57 pm

Re: Splitting a file after a new line

Post by tarja311 »

Darn, i was afraid of that. I can't believe for something as simple as this i am just so completely stumped on. I've been screwing around with this script all day. Oy! :(
User avatar
pickle
Briney Mod
Posts: 6445
Joined: Mon Jan 19, 2004 6:11 pm
Location: 53.01N x 112.48W
Contact:

Re: Splitting a file after a new line

Post by pickle »

Whenever I get stumped I stop trying to figure out the problem exactly, and plan out the general idea. It's a bit like writing an outline for an essay before writing the actual essay - it helps you get the layout of the problem squared away without getting distracted by the details.
Real programmers don't comment their code. If it was hard to write, it should be hard to understand.
Post Reply