This is highly ridiculous. HTTP URLs are not mandated to be case-sensitive (though it's recommended), and clearly lots of sites use them in a case-insensitive way. Robots should either consider robots.txt in a case-insensitive way (even if I'm conscious that lot of them, including major ones, currently don't do that, which is precisely what I consider to be a problem, and this is supported by what happened here -- where google is risking to appear as a fool). The following article has perfectly good arguments in favor of case-insensitivity or even more clever handling: http://www.slicksurface.com/blog/2007-04/be-careful-robotstx...
Also; failing to respect conditions of use of a service because an automatic process is not safe enough is not a completely exonerating excuse for the operator of such process...
An HTTP client must always assume URL paths and query strings are case-sensitive. You can't rely on the bing.com web servers always returning the same resource for http://www.bing.com/search?q=test and http://www.bing.com/Search?q=test just because the default filesystem for their OS is case-insensitive today. Actually it's unlikely the main entry point for their search engine is a file named "search" in some top-level directory. The only way for an author to tell you many URLs are equivalent is to use "301 Moved Permanently" responses to the canonical one from all the others, and Bing doesn't do that.
This is highly ridiculous. HTTP URLs are not mandated to be case-sensitive (though it's recommended), and clearly lots of sites use them in a case-insensitive way. Robots should either consider robots.txt in a case-insensitive way (even if I'm conscious that lot of them, including major ones, currently don't do that, which is precisely what I consider to be a problem, and this is supported by what happened here -- where google is risking to appear as a fool). The following article has perfectly good arguments in favor of case-insensitivity or even more clever handling: http://www.slicksurface.com/blog/2007-04/be-careful-robotstx...
Also; failing to respect conditions of use of a service because an automatic process is not safe enough is not a completely exonerating excuse for the operator of such process...