I currently have PhpDig however I have the overwhelming criticism that I must manually enter URLs of pages I want it to crawl.
Unless they have since updated it (I have 1.8.7 and they have gotten to 1.8.9 since in roughly two years I think) I'd like to find another script for my site that I can like PhpDig crawl my site from my own site however I would like to only have to enter in my own site's URL or a very minimal number of URLS. I'd hardly call a spider that only indexes manually requested URLs as a spider then I would merely an indexer. NO I know some people will say just use Google/Yahoo/MS, I'd like to have my own independent solution. Also by default I want it to obey robots.txt of course though I presume most people would think I'm wasting my breathe even mentioning this. Thanks for your recommendations!
Recomendations for on-sight local search/spider?
Moderator: General Moderators
- JAB Creations
- DevNet Resident
- Posts: 2341
- Joined: Thu Jan 13, 2005 6:44 pm
- Location: Sarasota Florida
- Contact: