Recomendations for on-sight local search/spider?

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

Post Reply
User avatar
JAB Creations
DevNet Resident
Posts: 2341
Joined: Thu Jan 13, 2005 6:44 pm
Location: Sarasota Florida
Contact:

Recomendations for on-sight local search/spider?

Post by JAB Creations »

I currently have PhpDig however I have the overwhelming criticism that I must manually enter URLs of pages I want it to crawl.

Unless they have since updated it (I have 1.8.7 and they have gotten to 1.8.9 since in roughly two years I think) I'd like to find another script for my site that I can like PhpDig crawl my site from my own site however I would like to only have to enter in my own site's URL or a very minimal number of URLS. I'd hardly call a spider that only indexes manually requested URLs as a spider then I would merely an indexer. NO I know some people will say just use Google/Yahoo/MS, I'd like to have my own independent solution. Also by default I want it to obey robots.txt of course though I presume most people would think I'm wasting my breathe even mentioning this. Thanks for your recommendations!
Post Reply