Page 1 of 1

Recomendations for on-sight local search/spider?

Posted: Wed Nov 07, 2007 6:50 am
by JAB Creations
I currently have PhpDig however I have the overwhelming criticism that I must manually enter URLs of pages I want it to crawl.

Unless they have since updated it (I have 1.8.7 and they have gotten to 1.8.9 since in roughly two years I think) I'd like to find another script for my site that I can like PhpDig crawl my site from my own site however I would like to only have to enter in my own site's URL or a very minimal number of URLS. I'd hardly call a spider that only indexes manually requested URLs as a spider then I would merely an indexer. NO I know some people will say just use Google/Yahoo/MS, I'd like to have my own independent solution. Also by default I want it to obey robots.txt of course though I presume most people would think I'm wasting my breathe even mentioning this. Thanks for your recommendations!