Page 1 of 1

I need a FIFO method of FOPEN, to pop the first line ....

Posted: Sat Feb 11, 2006 4:49 pm
by jclarkkent2003
Hi again, heh... This is a REALLY Stupid question and hella easy I know, I just can't remeber what is the MOST efficient method of doing this....

I need to open a list.txt file and pop the first line of it into a buffer.txt file, and then when the script is finished, it finds that line in the buffer and moves to the finished.txt file.

I will be running the same script multiple instances of ( run it 20 times at any given time so the buffer.txt will have 20 lines which i will also need to know how to search for and remove that line from the file.)

The list.txt could be 1 mb in size and I prefer not to load that into the memory, but if it's possible to just pop the first line into the buffer, and not have to open the entire file, recurse though each line checking if it's the right line ?

What's an efficient way to do this ?

Thanks~

Posted: Sat Feb 11, 2006 5:21 pm
by josh
If you're not appending to the file, the only way to modify it would be to load the entire file into memory. You can however read the file into memory one line at a time or use fseek() to jump to a certain byte inside of the file.

Posted: Sat Feb 11, 2006 5:30 pm
by jclarkkent2003
right now my method is this:

Code: Select all

<?

$filename = file("list.txt");
$first_line = array_shift($filename);

$handle = fopen{"list.txt","w+");
fwrite($handle,$filename);
fclose($handle);

$handle = fopen{"buffer.txt","a+");
fwrite($handle,$first_line);
fclose($handle);


and then when the file is done running....

$filename = file("buffer.txt");

for($k=0;$k<sizeof($filename);$k++)
{

if($filename[$k] == $firstline) { unset($filename[$k]); // not sure if this one will work for a variable inside an array , never tried. }

}

$buffer_removed_line = implode("\n",$filename);

$handle = fopen{"buffer.txt","w+");
fwrite($handle,$buffer_removed_line);
fclose($handle);


$handle = fopen{"finished.txt","a+");
fwrite($handle,$firstline);
fclose($handle);

?>
K, i didn't test the above but it's my raw coding for what I need to do. Is there a more efficient way, as like i said, say the list.txt can be upto 100 megs in size, and the buffer.txt be 50,000 lines, and the finished.txt be 300 mb

What is a more efficient way of doing this?

Posted: Sat Feb 11, 2006 5:31 pm
by jclarkkent2003
jshpro2 wrote:If you're not appending to the file, the only way to modify it would be to load the entire file into memory. You can however read the file into memory one line at a time or use fseek() to jump to a certain byte inside of the file.
See, Fseek would be perfect, but how can I then remove that line from list.txt ? I ONLY need to appened the FIRST line somehow the easiest fastest and most efficient way possible....

Posted: Sun Feb 12, 2006 3:56 pm
by jclarkkent2003
any ideas my brothers/sista's?

Efficiency is going to be a BIG deal with this script once it's more complete. Just two things left, this one to make it more efficient and the other one is my get page through proxy, then it's 90% complete.

Thanks in advance~

Posted: Sun Feb 12, 2006 4:12 pm
by feyd
use a database.

There is no simple and efficient way of FIFO'ing a large file.

Posted: Sun Feb 12, 2006 4:45 pm
by jclarkkent2003
yuck!!! lol, i want to stay clear of a database.

So your saying there is NO possible function or method I can fseek the first line, and somehow remove that first line as well without opening the full file ?

:(

So my method above is the ONLY possible way to do it? ...

Involving a mysql database would eat up the load like crazy and put strain on the server for the tasks i'm planning for this script. flat file db is alot easier on the server as there would be way too many COMPLEX mysql queries going... Mysql is already handling enough as is, I hate it for sake of shooting my server's load sky high ( that's what you get when you host thousands of sites on one server.)

Posted: Sun Feb 12, 2006 4:56 pm
by josh
mysql is generally going to be more cost efficient then reading the entire file into memory and then writing it back again. I recommend running benchmarks on both methods

Posted: Sun Feb 12, 2006 5:36 pm
by jclarkkent2003
oh i have ran benchmarks, heh, flat file is barely touches my load where as mysql eats it up like crazy until it's done, like i said, it's like that because there's aLREADY a few thosuand sites on the cpanel server using the mysql databases, mysql can only handle so much and it's there.