ToDotNet

With Fiddler (available at www.fiddlertool.com), you can create web requests and analyze the result in very good detail. It’s especially useful in this case, as you can create a request specific for certain user-agents. So download the tool and setup your ASP.NET 2.0 site on an IIS 6.0 environment. One thing to note as well, is that this site needs to be running under its own hostheader, not as a virtual directory.
Once installed, you take your web browser and go to

http://localsitename/default.aspx

The page default.aspx will be rewritten as page.aspx?ID=1 and everything works just fine.

Now, open up Fiddler and create the following request:

Accept: */*
Accept-Encoding: gzip, x-gzip
User-Agent: Mozilla/4.0

Set the url to

http://localsitename/default.aspx

and hit Execute. You should get status code 200, meaning OK. Now set the url to

http://localsitename/justafolder/default.aspx

and after you hit OK, again, you will get a 200 code. No problems so far.

Now, change the request to

User-Agent: Mozilla/5.0
instead of User-Agent: Mozilla/4.0

Hit Execute and bang… error 500, indicating an application error.
Here’s a list of user-agent entries that will result in an error:

Mozilla/1.0
Mozilla/2.0
Mozilla/5.0
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)
Mozilla/5.0 (compatible; Konqueror/.... (Tous les users agent de Konqueror que j'ai testés plantent)
Etc...

Some funny details:
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1 <= no error
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.0.1) <= error 500!

Ok, so let’s try to explain what happens. If you call RewritePath with the rebaseClientPath parameter set to “True”, the virtual path is reset. So why set it to False? Well, the setting of rebaseClientPath affects the action-tag of a form.

If I have an url http://mysite/myfolder/mypage.aspx which is rewritten to http://mysite/page.aspx?id=mypage, the form tag will we set as follows.

With rebaseClientPath set to true:

<form name="form1" method="post" action="page.aspx?ID=mapage" id="form1">

But with rebaseClientPath set to false:

<form name="form1" method="post" action="../page.aspx?ID=mapage" id="form1">

In case of a postback, the action in the latter situation (with rebaseClientPath set to false) is correct. Not in the first instance, because there is no page.aspx in the subfolder /myfolder.

Now, a workaround would be to manually set the UrlPostback to the correct location, but the ramafications are significant, and may affect the execution of the javascript manipulation the UrlPostback on a number of browsers.

What’s really troublesome is that it affects only the production version of a website. It’s not visible during development (usually you don’t let searchengines index your development and test environment, right?). Also, it happens only under IIS 6.0, using ASP.NET 2.0. And the only variable in this case is the user agent with which the site is accessed.

Now we know what the issue is, how do we resolve it? Well, doing it yourself is not easy and without risk, but here goes.

It has to do with the capabilities of the user-agent. When the site is hit by a user-agent having Mozilla/5.0 in the string, ASP.NET will be using System.Web.UI.Html32TextWriter. I’ve traced the bug with Intercept Studio and can confirm this. If you use another user-agent, for example Mozilla/4.0, System.Web.UI.HtmlTextWriter will be used. This is called adaptive rendering, which can lead to weird behavior.

So the problem lies in Html32TextWriter, but it’s unclear where this goes wrong exactly. The exception is thrown in System.Web.Util.UrlPath.ReduceVirtualPath(), but by the time the stack reaches this method, we’re already six method calls away from the last usage of Html32TextWriter (being used in System.Web.UI.HtmlControls.HtmlForm.RenderAttributes). If you have time to walk the stack with Reflector, go ahead. There are about 23 method calls in all.

You may want to wait for a fix to come from Microsoft, but if you can’t wait that long, there’s a hack solution to try for yourself. Say for instance you want to fix the issue with the Yahoo searchbot, Yahoo! Slurp (you would need to apply this to all user-agents in a similar fashion.)

Since Visual Studio 2005 we have the capability to create .browser files. These .browser files contain a definition of the capabilities of browsers. In your web project, add a folder called “App_Browser” and create a new file (i.e.: yahooslurp.browser). In this file, you put

If you want to debate the issue, please go to the thread at the Community Server forums.

PS: Why did I write this post? Well, the entire explanation is written in French, and not everyone can read this. Also, Nix and I know each other and he’s been helpful on a number of occasions, so if there’s anything I can do in return, like explaining the issue in English, I’m more than happy to help. Finally, this problem needs a proper solution, and the sites and servers I host can be just as much affected. I was fortunate not to have too much url rewriting in my sites, but that doesn’t affect the scope of the issue.