Recomendations for on-sight local search/spider?
Posted: Wed Nov 07, 2007 6:50 am
I currently have PhpDig however I have the overwhelming criticism that I must manually enter URLs of pages I want it to crawl.
Unless they have since updated it (I have 1.8.7 and they have gotten to 1.8.9 since in roughly two years I think) I'd like to find another script for my site that I can like PhpDig crawl my site from my own site however I would like to only have to enter in my own site's URL or a very minimal number of URLS. I'd hardly call a spider that only indexes manually requested URLs as a spider then I would merely an indexer. NO I know some people will say just use Google/Yahoo/MS, I'd like to have my own independent solution. Also by default I want it to obey robots.txt of course though I presume most people would think I'm wasting my breathe even mentioning this. Thanks for your recommendations!
Unless they have since updated it (I have 1.8.7 and they have gotten to 1.8.9 since in roughly two years I think) I'd like to find another script for my site that I can like PhpDig crawl my site from my own site however I would like to only have to enter in my own site's URL or a very minimal number of URLS. I'd hardly call a spider that only indexes manually requested URLs as a spider then I would merely an indexer. NO I know some people will say just use Google/Yahoo/MS, I'd like to have my own independent solution. Also by default I want it to obey robots.txt of course though I presume most people would think I'm wasting my breathe even mentioning this. Thanks for your recommendations!