search engines and forums

[Vegan]

I spent much time over the last several weeks checking robots.txt for social media and forums. No surprise is that most block search engines from crawling them.

This forum is crawled by Google at al so as a result any solutions here are easily found and it cuts down on the same old post surfacing over and over again. While forums etc have local search the lack of skill makes it hard to leverage.

Not many forums allow search but some do. I suggest forums based on phpBB use a careful robots.txt that blocks user profiles but allow reading posts.


I noticed it took Google more than 90 days to discover my posts here. This suggests it may be some time before this forum comes back to life. There is a lot of good content so hopefully it bounces back.

Bing has been better at discovering backlinks but yet content crawling is really slow. After 90 days it has not even discovered 300 pages of some 6000 or so on my site.