Robots.txt handling via a HTTP Handler in asp.net

Within IIS you need the following lines of code:

Withing the node of web.config section of web.config you need the next 2 lines in the correct order:

<verb="GET" path="/Robots.txt" type="HandlerExample.MyHttpHandler">- this one handles the robots.txt file 
<add path="*.txt" verb="GET" type="System.Web.StaticFileHandler" /> - this one handles other .txt files and serves them corectly

Within a class you will need the code to handle the robots file that is written out dynamically.

Getting IIS to Pass .txt Requests to ASP.NET

At the moment, IIS is still serving the txt files instead of letting .NET know about them – the request for a .txt file doesn’t even reach .NET. Lets tell IIS to pass that request through to .NET.

Get the path to the ASPX engine

1. Open IIS and right click on your website and bring up the properties screen
2. Go to Home Directory > Configuration. You will be on the Mappings Tab.
3. Locate the ASPX item and click Edit – Copy the path in the Executable Field and cancel out of that window. We don’t want to change anything there – we just want the path

Create the ISAPI Entry for .txt

You are still on the Mappings Tab for your site in IIS 1. Click “Add” 2. Populate the Executable path with the value you copied in the last section 3. Enter “GET” in the “Limit To” field 4. Untick the box "Verify that file exists". Press OK a million times to save all your changes. You’re done with IIS.

Check the result

When you now try to browse to your text file you should get a blank page. This means IIS is now passing the request to ASP.NET, but ASP.NET doesn’t know what to do with it.