Crawling an entire site

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

User avatar
anjanesh
DevNet Resident
Posts: 1679
Joined: Sat Dec 06, 2003 9:52 pm
Location: Mumbai, India

Post by anjanesh »

My code actually writes to a file only. After writing it echos 'Done';
Problem is I've no way to figure out if somethings gone wrong in the middle (like infinte loop).
For that I need something to show that some data is actually being retrieved.
I may even know the number of iterations required - so I can cross check if '|' is displayed more than the required times.

Code: Select all

...
Loop
 {
 	// Retrieve details - This actually writes to a file
 	fwrite($handle,$details);
 	flush();ob_flush();
 	echo "|";
 	usleep(100); // To avoid taking entire CPU usage
 }
echo "Done !":
BTW, will flush() have the effect of stopping the CPU usage until it outputs to browser so far ? And then resume back. This way CPU isnt 100% used. Does flush() work like this ?
Post Reply