robots.txt how prevent pages under website's root folder fro
Posted: Thu Jun 03, 2010 8:36 am
Hello
I have this question. Let's say I have a domain called: www.domainx.com
Let's say that:
- I have an htaccess rewrite (hidden redirect) in order to display the website stored under the folder /websitex whenever an http request gets www.domainx.com That is to say, if you go to www.domainx.com the url will not change. However, the content displayed in the browser will be really the one stored under the folder called "websitex". In other words, the browser displays the same content as if you had typed www.domainx.com/websitex
- I do not want search engines to index the contents under the root folder of www.domainx.com. I just want them to index www.domainx.com and the contents under www.domainx.com/websitex
Is there any way to achieve this through the use of a robots file?
I would try something like this but I am not sure:
User-Agent: *
Disallow: /
Allow: /websitex
My concern is that I am not sure whether this would prevent www.domainx.com from being listed by search engines or not. Actually I would like www.domainx.com to be listed by search engines.
Any idea?
Thanks
I have this question. Let's say I have a domain called: www.domainx.com
Let's say that:
- I have an htaccess rewrite (hidden redirect) in order to display the website stored under the folder /websitex whenever an http request gets www.domainx.com That is to say, if you go to www.domainx.com the url will not change. However, the content displayed in the browser will be really the one stored under the folder called "websitex". In other words, the browser displays the same content as if you had typed www.domainx.com/websitex
- I do not want search engines to index the contents under the root folder of www.domainx.com. I just want them to index www.domainx.com and the contents under www.domainx.com/websitex
Is there any way to achieve this through the use of a robots file?
I would try something like this but I am not sure:
User-Agent: *
Disallow: /
Allow: /websitex
My concern is that I am not sure whether this would prevent www.domainx.com from being listed by search engines or not. Actually I would like www.domainx.com to be listed by search engines.
Any idea?
Thanks