Search Engines and custom 404 pages

XML, Perl, Python, and other languages can be discussed here, even if it isn't PHP (We might forgive you).

Moderator: General Moderators

Post Reply
User avatar
Skittlewidth
Forum Contributor
Posts: 389
Joined: Wed Nov 06, 2002 9:18 am
Location: Kent, UK

Search Engines and custom 404 pages

Post by Skittlewidth »

Just interested if anyone has any thoughts on this little scenario - I've coded it in .NET but the theory should be the same regardless of the language.

An SEO company ripped a website this company made for Search Engine compatibility. I've seen it happen before, SEOs are in the great position of being able to find fault with what ever site you build and not have to deal with coding the solution so I'm not worried by that.

Anyway, to resolve one of the issues I used ISAPI_ReWrite (like Mod_ReWrite) to give the urls more friendly names and erradicate the query string. I tried a variety of regular expressions and things to try and pass the query string data in a more user friendly format, but at the end of the day the client simply wanted http://www.mydomain.com/blah to map to pages like index.aspx?id=x so we ended up writing a stupidly long hardcoded httpd file.

The problem it left us with was that if they ever added a new page and gave it a "friendly name" in the CMS, it wouldn't exist in the httpd file. I've used PHP to dynamically add ModReWrite directives to a file before but I wasn't sure how much of a good idea that was security wise.

Anyway the idea I came up with today was to put a custom 404 error page on the test domain with some code to check the requested url and see if it does infact exist as a page in the database if it doesn't appear in the httpd file. If it does it uses Server.Transfer to seamlessly pass the user to the correct index.aspx?p=x in the background without showing the user the real url in the location bar. If it doesn't exist it serves a "Page Does Not Exist" message.

The experiment worked, and the end user wouldn't know any different. But what will a Search Engine think when it comes across such a page? Since something has to have detected a 404 scenario to have been sent to that custom page in the first place, would a Search Engine spider just stop there, or would it continue to read the resulting page if one is found?
Post Reply