Page 1 of 1
Url spider
Posted: Mon Nov 28, 2005 9:21 am
by wizzard
Hello,
I have made a little exchange script for my website and i work with a spider to check the website from the exchanging members. If my link is not on their link page it disabled the link.
Now i have a problem when someone is putting a link on his site but the link is placed on page 2,3,... then my spider is not finding the link.
Is there a solution for this?
Re: Url spider
Posted: Mon Nov 28, 2005 9:23 am
by foobar
wizzard wrote:
Is there a solution for this?
Follow URLs that point to the same site until you find the link. If you haven't, you don't link back.
Posted: Mon Nov 28, 2005 9:41 am
by wizzard
Well the problem is it the links on some partner sites are limited by 20 per page and my link is then on the second, ... page
Is it possible to let my spider auto browse thru the pages?
Posted: Mon Nov 28, 2005 9:41 am
by Burrito
if it's a "true" spider...it should crawl all of the pages.
Re: Url spider
Posted: Mon Nov 28, 2005 9:49 am
by Weirdan
wizzard wrote:
I have made a little exchange script for my website and i work with a spider to check the website from the exchanging members. If my link is not on their link page it disabled the link.
I would approach this task in a bit different way: register visitors from the sites exchanging links with you. When there's no single visitor for a week (or so) disable the link to that site. Once a visitor has arrived from the site, turn the link back on.
This approach eliminates the need for the spider altogether and keeps you 'safe' from abuses like putting the link to your site in invisible div.