Building a Web API 2 project from scratch using OWIN/Katana .NET Part 4: async controllers and mini-DDD


In the previous post we briefly looked at a new hosting project by Microsoft called Helios. It is meant to be the future of web application deployment where Helios removes the necessity of having the entire System.Web dependency in your web project. We saw that Helios is only in a preview state so it shouldn’t be used for real-life projects yet.

In this post we’ll diverge from OWIN/Katana and instead see how we can add asynchronous controller methods to our current CustomersApi web project. We’ll also build a miniature version of a layered application. We’ll put the layers into separate folders for simplicity.

Read more of this post

Web farms in .NET and IIS using Web Farm Framework 2.2 part 3: deployment and tests

In this blog post we’ll put into practice some of the things we’ve gone through in this series on web farms. In particular we’ll take a closer look into the following scenarios:

  • How to deploy a web site to a WFF web farm with ARR as the load balancer
  • How to set up the state service common to all web farm machines
  • How to set up the database state management common to all web farm machines


The first thing we want to test is how to deploy our website to the web farm. I’ve prepared a very simple ASP.NET web forms project where the Index page prints out the machine name. I wanted to show some more-or-less unique ID on the screen which shows that we’re getting the response from different machines as the load balancer distributes the web requests. I wanted to make this ID independent of the web site settings as all such settings should be identical throughout the farm. Recall that the primary server is the “leader”, so whatever is deployed on that machine will also be replicated on the secondary servers.

I show the machine name in a label control as follows:

protected void Page_Load(object sender, EventArgs e)
	if (!IsPostBack)
		lblUniqueId.Text = GetUniqueIdentifier();

private string GetUniqueIdentifier()
        return System.Environment.MachineName;

The ID is shown on the index page:

Show machine name on screen

Next I’ll need to create the correct bindings in IIS. Follow these steps:

  1. Log onto the primary server of the web farm
  2. Open the IIS manager
  3. Delete the existing Default Web Site in the Sites folder
  4. Create a new website: give it some name – I called mine wfftest.apica.local. Leave the host name blank to keep it simple
  5. Create a folder for the deployment package in the Physical path text box

Change the application pool settings as follows:

Change app pool settings on primary server

Normally you would have to go through this procedure on every machine of the web farm. However, we have WFF in place, so I’ll log on to one of the secondary servers to see if the changes have been propagated:

Web site changes propagated to secondary server

I’ll now deploy the web page to the web site folder using the Web Deploy technique we saw in this post. I don’t know about you but I got an exception at first:

Web deploy failed

Web deployment task failed: make sure web deploy is installed and the required process – Web Management Service – is started.

I’ll show here all the steps I had to take to make this work:

1.) Select the Machine in IIS on the primary server and see if the below icons are available:

Management service icons in IIS

If not then you’ll need to enable them. Go to Start, Administrative Tools, Server Manager. Select Roles, Add Role Services and click the Management Service feature:

Add management service to IIS

Click Next and Install. After this refresh the IIS manager window and you should see the three icons that were missing.

2.) Start the service:

Double click the Management Service icon in IIS. Make sure you that remote connections are enabled and you allow for a mixture of credentials. Leave the port number at 8172. Click Start after these changes:

Start management service

3.) Check your firewall settings:

The following inbound rule must be enabled in the Windows Firewall manager (Start, Administrative tools, Windows Firewall and Advanced Security):

Management service inbound rule enabled

4.) Install Web deploy:

This is easy with WFF. Log onto the controller machine, select Platform provisioning and select Web Deployment Tool 2.1:

Web deployment tool with provisioning

Check that the tool is deployed on the primary server:

Web deployment tool installed successfully

5.) Check if web deployment handler is installed:

In the Programs and Features window shown above right click Microsoft Web Deploy and select Change. Make sure that all features are enabled. You’ll see a red X if they are disabled. Select the following option:

Install web deploy handler

6.) Restart the web management service:

Restart web management service

In case none of the above options help then check out the management service log file.

The relevant log file is available on the primary web farm machine under c:/inetpub/logs/WMSvc. There may be an additional folder there, such as W3SVC1. Within that folder you will find a text file, open that. You’ll see some cryptic messages such as:

2013-06-27 13:51:58 HEAD /msdeploy.axd site=wfftest.apica.local 8172 – – 404 7 0 234

This means that msdeploy.axd was missing which in turn means that WebDeploy wasn’t installed. Check the numeric codes at the end – I know that due to the IIS response: 404.7. You can find all IIS response codes and subcodes here.

Another problem you may encounter is that IIS doesn’t allow some file extensions. Check the applicationHost.config file on the primary server under c:/windows/system32/inetsrv/config/applicationHost.config. Look for the element called fileExtensions. It has an attribute called allowUnlisted. Set it to true if it’s set to false.

So, after all this I’ve been able to deploy according to VS:

Web deploy success in Visual Studio

Check that the package has been deployed to the correct folder. In my case it has indeed:

Package deployed to correct folder

Check the secondary servers as well to see if the controller machine has updated them. You should see that everything is jolly well. We’re making progress! Let’s continue with the setup.

Log onto the controller machine and open the IIS manager. Delete the Default Web Site. Create a new site called arrbase pointing to a blank folder located anywhere you like on c:\. Leave the IP and port values untouched and the Host name field blank:

Create arr base on controller machine

The next step is to set up URL rewrite using ARR. Select the name of the server farm and click the Routing Rules icon:

Routing rules icon in IIS

In case you don’t see it you’ll need to install ARR – I had to install it as well. I assumed that installing WFF would be enough to use the capabilities of ARR but I was wrong. On the controller machine start Web Platform Installer and search for ARR:

Arr in web platform installer

Reopen the IIS manager and now you should be able to open Routing Rules. You’ll see that the Use URL Rewrite to inspect incoming requests checkbox is unchecked. Check it and apply the changes:

Set routing rule in ARR

Now we should be good to go. I re-named my web farm to wfftest.apica.local which is only available from within the company’s intranet where I work:


I asked our technicians to link this URL to the controller machine’s IP so that I can reach it from my machine. An alternative solution is to modify your hosts file at c:\windows\system32\drivers\etc\hosts. Open that file with Notepad using admin rights. Enter the IP of the controller machine and the web farm address, e.g. wfftest.apica.local. Make sure to put a tab in between the IP and the address.

So I enter this address in a web browser on my machine and… :

Wff test apica local responds

So I got a response from the primary machine in the web farm. I want to check if all 3 machines in the web farm work so I set the load balancing algorithm to round robin:

Set LB algorithm to round robin

…and then press F5 a couple of times. Apparently it takes a couple of seconds for the algorithm to be implemented but I get the response from the two secondary servers as well eventually:

Response from secondary server one

Response from secondary server two

Before we move on let’s re-test the deployment process and we’ll keep an eye on the controller machine. In the web project I add a new line under the “Hello from machine” bit just to be able to check that the changes had been deployed. Watch the controller machine carefully as the deployment process is rather quick:

Controller taking turns to deploy the web project

I then refresh the web page and the updates had indeed been deployed:

Web update successful

Session management

If you have a completely stateless web application, such as a REST web service, then you may as well stop here. You can add some health checks using the techniques shown here to ARR and then you are good to go. You have a functioning web farm with pretty decent provisioning and load balancing features for a relatively low price. If you use caching in your web service then you’ll need to look into some distributed cache solution. A bit more on that in the conclusion part of this post.

However, web sites are usually stateful. Lots of different data can be stored in session variables to carry them from one page to another. You can read about session management in a web farm in this post, here I only want to test two features outlined there: saving the session in an external state service and alternatively in a database. So make sure you read through that post as I will not repeat the details here. I need to make the model web site use the session state. So I create the following markup in Default.aspx within the MainContent placeholder:

Hello from machine <asp:Label Id="lblMachineName" runat="server" />
	<div>Enter your first name: <asp:TextBox runat="server" ID="txtFirstName"></asp:TextBox></div>
	<div>Enter your last name: <asp:TextBox runat="server" ID="txtLastName"></asp:TextBox></div>
	<asp:Button Text="Click me" ID="btnSubmit" OnClick="btnSubmit_Click" runat="server" />

The Submit button has the following click event:

protected void btnSubmit_Click(object sender, EventArgs e)
	Session["Firstname"] = txtFirstName.Text;
	Session["Lastname"] = txtLastName.Text;

I modified the HTML on About.aspx as follows:

	Hello from machine <asp:Label Id="lblMachineName" runat="server" />
		<asp:Label Text="" ID="lblFirstName" runat="server" /><br />
		<asp:Label Text="" ID="lblLastName" runat="server" />

…with the Page_Load event of About.aspx.cs looking like this:

protected void Page_Load(object sender, EventArgs e)
	if (!IsPostBack)
		lblFirstName.Text = Session["Firstname"] != null ? string.Concat("Your first name is: ",   Convert.ToString(Session["Firstname"])) : "There's no First name in the session!";
		lblLastName.Text = Session["Lastname"] != null ? string.Concat("Your last name is: ", Convert.ToString(Session["Lastname"]))	: "There's no Last name in the session!";
		lblMachineName.Text = System.Environment.MachineName;

I believe this is easy to follows. Let’s now deploy the project as it is without making changes to session handling and see what happens. So I fill in my first and last name and press the Click me button and… …I got an exception saying that validation of viewstate MAC has failed:

MAC validation failed

So we are required to change the session state management anyway. I took the following steps to set up a state service:

  1. Picked a development machine separate from the web farm machines but still within the same intranet
  2. Started the Windows service called ASP.NET State Service on that machine
  3. Changed the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aspnet_state\Parameters\AllowRemoteConnection value to 1 on that machine and restarted the state service
  4. Declared in web.config that session state must be handled through a state server using the address of the state server, which is the dev machine in my case I mentioned under point 1
  5. I inserted a machine key in web.config which will be common to all machines in the web farm

So I redeploy the web app and land on the primary machine. I enter my first and last name and press the button:

Enter first and last name on primary machine


Session state management on state server successful

That’s looking really good. We ended up on a different machine and the state is still available. Refresh the About page to land on different machines and you’ll see that the state is available.

OK, let’s try to save the session in a database now. I performed the following steps:

  1. Set up the ASPState table in a database available to all machines in the web farm using the aspnet_regsql.exe tool: aspnet_regsql.exe -ssadd -sstype p -S [database machine name] -U [db admin username] -P [db admin password]
  2. Check the database and I see the following table structure: ASPState table successfully created
  3. Update the sessionState tag in web.config (details below)

My sessionState element in web.config looks as follows:

<sessionState mode="SQLServer" allowCustomSqlDatabase="true" sqlConnectionString="ASPStateConnectionString">

…where ASPStateConnectionString refers to a connection string in web.config which has the following format:

<add name="ASPStateConnectionString" connectionString="Data Source=[DB machine name];Initial Catalog=ASPState;User ID=xxx;Password=xxxxx"
   providerName="System.Data.SqlClient" />

I deploy the web app as usual and perform the same test as above. It went well so I check the entries in the ASPState database:

ASPNet table session values

So we have successfully tested the most basic session management features for an IIS web farm.

A short addition: you can add an extra level of security to the database type of solution by synchronising your primary SQL server with a secondary one which serves as a failover solution. You can achieve this using the technology called SQL AlwaysOn. That way if your primary server fails then the sessions will be available from your failover server. If you find that the database solution is too slow but don’t want to take the risk of your state server going down then you can check out Memcached which is a free tool that offers a highly available short-term caching solution. An alternative solution is CouchBase.

Conclusion and possible improvements

We currently have the following model:

Blog WFF model

This is a good foundation but we still have a couple of vulnerability points:

  • ARR doesn’t provide high availability and scalability in itself – the solution is to set up an identical load balancing machine with ARR and NLB and install NLB on the existing ARR machine as well. Here‘s a good guide on how to proceed. The key idea is to provide a virtual IP for the two ARR+NLB machines where NLB is responsible for high availability
  • ARR and WFF are located on the same machine, but there’s nothing stopping you from putting them on two different machines: if this machine dies then WFF can still control the web farm
  • Set up another state DB machine and put SQL AlwaysOn in place in order to synchronise it with the main state DB
  • The WFF controller can still die and then there’s no content replication from the primary server to the secondary ones: at present this cannot be helped, i.e. there’s no high availability solution for the controller machine in WFF2.2 – your best bet is to monitor this machine externally and react quickly if it goes offline and avoid deploys during this time. However, even if it goes down your web farm will still be responding to web request – provided that the load balancer is located on a different machine
  • The primary server can of course also go down meaning you cannot deploy to it and WFF will not be able to replicate its content to the secondary servers – the solution is straightforward: just designate another primary server, check the screenshot below

To designate a secondary server as the primary one in the WFF GUI:

Designate a secondary server as the primary one

Right-click the secondary server name and select “Designate as Primary Server” from the context menu.

So if we implement the solutions suggested above to solve the weak points of our current architecture we may end up with something like this:

Wff finalised structure

This is a cost-effective high availability and high scalability system using well tested and reliable software solutions. You can add new web servers to your farm and then update the servers list on the ARR+NLB machines and on the WFF controller.


We haven’t talked much about caching yet. Say that you’d like to cache the results of the database query using one of the several built-in caching mechanisms in .NET: ObjectCache, HttpCache etc. If you cache the result on web farm machine A then that cached value will not be available on machine B automatically. Both machine A and B will have the cached value in their memory eventually if the web requests are routed to both of them, but their expiry date will be different. Maybe machine A will fetch a more recent result of the database query than machine B because machine B stores the value with a later expiry date. So essentially we’ll need a similar distributed solution for caching as for session management: the cached values should be available on separate machines that all web farm members have access to. We mentioned Memcached and CouchBase before in this post. They are both really good products with a .NET library.

CouchBase has an advantage over Memcached: in case you set up more than one external cache machine – a good idea if you need a failover solution here as well – then it’s desirable for the cache machines to synchronise their cache data as soon as there’s a change. We don’t want to the cache machines to return different objects for the same cache key, right? Memcached servers don’t talk to each other so you may actually end up with that scenario. CouchBase servers however provide synchronisation: say that web farm member A writes to the cache; this object with a certain cache key will be sent to the CouchBase server cluster. CouchBase will make sure that all cache machines in the cluster will have this updated value.

You can even use CouchBase to take care of session management for you if you don’t like the database solution outlined above. You replace the sessionState node in web.config to use an external session management solution. After all, session values are key-value pairs, much like cached objects.

Pre-compiled web package

One last improvement that I want to mentioned is pre-compiled websites. In the present solution if you deploy the web package then it must be compiled on the server when the first web request comes in after the deploy. This can cause a slight delay in the response time for the first visitor. It is possible with MSDeploy to pre-compile your web project before deploy. You can even automate that using your Continuous Integration server, such as TeamCity.

If you perform all the suggested improvements then you’ll end up with a distributed system which has the following characteristics:

  • High availability
  • High scalability
  • Distributed, highly available cache service
  • Distributed, highly available session state service
  • Automated content replication
  • Seamless deployment from the client’s point of view: the cache and the state are still available during and after a web deploy and the client doesn’t see a longer response time on the first web request

Web farms in .NET and IIS using Web Farm Framework 2.2 part 2: provisioning a server farm

By the end of the previous post we had a system with 4 machines up and running: a controller with WFF installed, a primary web farm server and two secondary web farm servers. We also saw an example of application synchronisation using WFF. In this post we’ll look at application and platform provisioning.

Content mirroring

By default all content of the primary server is mirrored to the secondary servers. The first thing I want to check is whether web sites and their folder content are copied over. Let’s log on to the primary server and create a new website. Don’t worry about the URL and the binding, I’ve created the following:

Create test website on primary server

A quick look at the trace messages on the controller shows the following:

Trace message after website creation

So far so good, our changes have been noted and apparently the sync process has started. I’ll make a couple of changes to the application pool to see if those changes are mirrored as well:

Changes to the application pool

I’ll also simulate a deploy and place an empty index.html file in the deployment folder I created on the primary server, c:\hello. I’ll now check the status on the secondary servers. I can see that all elements have been correctly mirrored:

Changes to website applied to secondsary server

Platform provisioning

With platform provisioning you can synchronise the secondary servers with the modules installed on the primary server. You can also select the interval between sync cycles. You can open the platform provisioning options by selecting the name of the server farm you created and then double-clicking the Platform Provisioning icon:

Open platform provisioning options

This opens the Platform Provisioning window:

Platform provisioning window

The purpose of the two checkboxes and the Sync interval textbox is easy to understand I believe. You can then select products to provision in the Additional Products section. At first this is confusing as the list will be empty and you may think there are no products available. You’ll need to double-click the product name row to bring up a list of products. I selected the PHP Manager for IIS package, but feel free to select whatever you want. Select Apply when you’re done:

Apply platform provisioning

Clicking Apply will bring up the list of products that the web platform installer will provision. Click I Accept and let’s see what happens. Select the Servers node in the Connections pane and you’ll see that synchronisation has started. The changes have been noted:

Handle configuration change status after applying provisioning

The servers are brought offline one by one:

Remove one server at a a time during platform provisioning

The first machine has been temporarily removed from the farm while the provisioning process takes place. You’ll see the related trace messages as well:

Trace messages during platform provisioning

Of course I want to verify that the selected product has been deployed to the secondary servers and it has indeed:

Platform provisioning successful

There’s another way to install products on the servers. You can even install the product on all machines of the web farm or just a selected one. If you want to install a product on the web farm then select the name of the web farm in the Connections pane on IIS on the controller machine and click the Install Products link:

Install products link at web farm level

If you wish to install a product on a specific machine then select one of the servers under the Servers node. You’ll find the same link as shown above under the Actions pane to the right. In either case the Install Products window will open where you can select the products to be installed:

Install products window

Application provisioning

This is similar to platform provisioning but it’s dedicated to Web Deploy. If you recall from the previous post WFF has a dependency on Web Deploy 2.0 so is is here where it is heavily used. You can open Application provisioning as follows:

Open application provisioning

The options look similar to what you saw under platform provisioning. You’ll need to double click the Provider cell to bring up all MSdeploy providers:

Application provisioning options

These are all MSDeploy providers, so you can automate MSDeploy commands. You can find the list of providers and what they do here and their settings here. The path will be the folder where the provider will be installed. You can open the settings, i.e. the MSDeploy program arguments by clicking the ellipsis:

Open provider settings MSDeploy

Server operations

You can perform operations on the server farm or individual machines in the farm. The term operation is quite wide in this case and can be grouped into 4 categories:

  • Provisioning: besides the types of provisioning mentioned above you can e.g. update Windows or get a list of installed products
  • Windows services: you can control the Windows services installed on the target machine(s), change their state and reboot them.
  • Diagnostics: get a list of processes and requests running on the machine(s)
  • Server: test your servers

Let’s look at a couple of examples. These are all available on the controller machine. If you’d like to perform operations on the web farm level then click the Server Farm Operations link:

Open server farm operations

You can find a similar link if you open the web farm node and select one of the machines:

Open operations on a single server

The link will open the Server Operations window where you’ll recognise the categories from the list above:

Server farm operations window

Click on the categories and subcategories to open the parameters you can provide to the operation. Most of them are quite self-explanatory, such as the Windows Update or Install Products option. Click the ‘Run’ button when you’re done and the results of the operation will be shown in the small panel in the bottom. The results will be presented in an XML format. The Query Installed Products produces an output similar to the following:

Query installed products operations result

It’s not easy to look at the result in that small panel. If you want to view the details then copy and paste it to a text editor.

You can get the processes and requests running on the server under the Diagnostics and monitoring section. You can leave the filtering text boxes empty if you want to retrieve a full list:

Get processes operations result

This gives you the list of processes as you can see them in the Task Manager serialised into XML.

In the next post we’ll see how to deploy a real web site in the sample web farm.

Web farms in .NET and IIS using Web Farm Framework 2.2 part 1: how to set up a web farm using WFF

So far in the series on web farms we’ve looked at two concrete solutions for load balancing: Network Load Balancing and Application Request Routing. Both are software based solutions that are relatively easy to use and get started with. However, there’s still a certain amount of manual work you need to perform: add new servers, install applications, install load balancers etc.

There are frameworks out there that can take the burden of manual installations and settings off of your shoulder. However, they can be quite complex with a steep learning curve. Also, as there is much automation involved it’s important to understand them inside out.

Microsoft provides two frameworks as mentioned in the first part of the series on web farms: the Web Farm Framework (WFF) and Windows Azure Services. As the title of this post suggests we’ll look at WFF a bit more closely. The latest version is WFF2.2 at the time of writing this post.

Note: if you are completely new to web farms in IIS and .NET then start here. Some concepts may be difficult to understand if you lack the basics.


WFF is a module add-on to IIS. WFF uses the Web Deploy module to update products and applications. Here you’ll find the homepage of WFF2.0 but we’ll concentrate on WFF2.2 which is the most up-to-date version at the time of writing this post. Some of the more interesting features include the following:

  • Server provisioning: often called elastic scale, WFF keeps track of all product installations on the server so that you can provision new servers any time with almost no effort. The process takes a server with only the operating system installed and WFF brings it up to the same status as the existing web servers
  • Application provisioning: WFF can perform application installations in each machine in the web farm one by one. It takes the server out of rotation, installs the components, brings the server back online, and then continues with the next server. It’s possible to roll out applications using Web Platform Installer or Web Deploy
  • Content provisioning: WFF also manages application content provisioning and updates
  • Load balancing support: WFF has built-in support for Application Request Routing (ARR) and third party load balancers so it can manage servers and sites for you
  • Log consolidation and statistics: WFF offers status and trace logs from all servers in the web farm
  • Extensibility: WFF is highly configurable and allows you to integrate your own applications and processes into WFF
  • API: WFF can be accessed programmatically using the Microsoft.Web.Farm namespace. It can also be managed through PowerShell commands.

It’s important to stress that WFF is not a load balancer in itself but works in unison with load balancers. The default load balancer that WFF works with is ARR.

The disadvantage of WFF is that it introduces an extra point of failure: it depends on a single controller node. If that fails then WFF will be unable to monitor or manage the server farm until it is back online again. If you make a change to your primary server then that change won’t be propagated to the secondary servers. Your web farm will still continue to work, i.e. your clients will still get web responses from your website but WFF will not be able to synchronise between your server farm machines to keep them up to date. This shortcoming may be corrected in a future version. Read on to find out the role of each server type in a WFF setup.

The starting point

The plan is to start the investigation from scratch. I now have 4 clean machines at my disposal to test WFF: one controller machine, one primary server and two secondary servers. Every machine has Windows Server 2008 R2 and IIS7.5 installed and they all have a 64-bit architecture. There’s no website deployed on any of them yet. All machines sit within the same network so that they can communicate with each other. Don’t make your life harder by putting one machine in the cloud and the other in the local network of your company. Make the start as easy as possible.

The roles of each machine are as follows:

  • Controller: the WFF controller server which is the controller of the web farm. This is the only machine that needs to have WFF installed. It monitors the status of the web farm machines, provides synchronisation, provisions applications, etc., it is a real workhorse
  • Primary server: this is the blueprint machine for all web farm machines. The changes made on this machine will be propagated to the secondary servers.
  • Secondary servers: the “slaves” of the primary server. They will be updated according to the changes on the primary server.

The secondary servers must be of the same architecture as the primary server, i.e. either 32 or 64 bit. You must have an administrator account to the machines.

Setting up the controller

So now I’ve logged on to the controller machine. Installing WFF is at present not as straightforward as I initially thought. I expected that the latest Web Platform Installer, which is in version 4.5 at this date, will be enough to be able to select WFF and install it with all the dependencies. However, WPI 4.5 won’t have that product listed at all.

Instead you’ll need to install v3 of the web platform installer available here. Install it and then download v2.2 of WFF from here. Select the correct package depending on the architecture on the next screen:

Select WFF architecture

When the installation starts you may be greeted with the following message:

Web deploy is a prerequisite

DON’T go to the link shown on the little popup. It will lead you to the Web Deploy v3 download page. It will also update your freshly acquired WPI v3 to v4.5, arghh! As it turns out we’ll need Web Deploy 2 instead.

You can download Web Deploy 2.0 from this link. Select the correct package according to the architecture of the machine and step through the very simple installer.

So now you should have the following platforms available among the applications on the controller machine:

Correct WPI and Web Deploy packages

Now go back to the WFF 2.2 download page and try to install WFF again. Hopefully you’ll get the same message as me:

Successful installation of WFF 2.2

Yaaay! I don’t really understand why this process is so cumbersome. It’s as if Microsoft want to hide this product. If you happen to know the reason then feel free to comment.

The controller machine will have the main instance of WFF in the network. WFF will install a WFF agent on the web farm machines.

Setting up the secondary servers

As mentioned above you should have an admin account to the servers. This account will be used when we create the web farm in a later step. We also need to make a couple of exceptions in the firewall. Open the Control Panel and select the System and Security group:

Open control panel

Then select the option marked below:

Allow program through firewall option

You’ll need to open up 3 channels. They are Core networking, File and printer sharing and Remote administration:

Select 3 options in allow programs through firewall window

Do this on all secondary servers in the web farm.

Create a server farm

These steps should be performed on the controller server where we installed WFF 2.2. Open up the IIS manager and you should see a new node called Server Farms:

Server farms node in IIS

Right-click that node and select Create Server Farm… This will open the Create Server Farm window. The first step is to give the web farm some name and provide the administrator account. Don’t worry too much about the name, you can always change it later. Make sure to select the Provision server farm option:

Specify server farm details

The Server farm is available for load balancing option will set up Application Request Routing (ARR) as the load balancer. Uncheck this textbox if you have a different load balancing option in place.

Click next to go to the Add Servers window. Here you can add all the servers participating in the server farm. I’ll first add the primary server so I enter the primary server machine name, check the “Primary Server” checkbox and click add. When you click add then the wizard will attempt to connect to the server using the admin account you’ve provided. Upon successful setup the primary server should be added to the list of servers:

Add primary server in WFF

Then add the secondary servers the same way. If there’s a communication error then you’ll see an error message: maybe the machine name is not spelled correctly, or there’s a firewall problem. At the end of the process I have 3 machines: 1 primary and 2 secondary:

Server farm machines ready

Click finish and there you are, you’ve just set up a web farm using WFF! Open the Server Farms node and you’ll see the farm you’ve just created.

There are a couple of things going on right after the setup process. WFF will install the WFF agents on the primary and secondary servers and it will also synchronise all content, applications, configurations etc. between the primary and secondary servers. Select the Servers node under the farm you’ve just created. This will open up the Servers section in the middle of the screen. At first you may see No under Ready for load balancing:

Ready for load balancing

Here it already says yes as I forgot to save the picture just after the farm creation, but it should say no at first. Look at the bottom half of the screen. You will see that WFF starts the synchornisation process right away:

Wff trace messages

After a couple of minutes the ‘no’ should go over to a ‘yes’.

Testing the primary server sync

Let’s now test if an application is copied from the primary server to the secondary ones. Log onto the primary server and download WPI v3 like we did above. Using Web PI install Url Rewrite 2.0. Go back to the controller machine and watch the status of each web farm member. You’ll see that each is temporarily removed from the network while updating the member machine:

WFF synch machine 1

The machine is brought back online and then it’s the second machine’s turn to be updated. I’ll now log onto the secondary machines to see if the update really happened, and it actually did:

Sync successful with WFF

So far so good!

Adding and removing servers are straightforward operations. Just select the following links in the Actions pane on the right:

Links to add and remove servers

You can see the WFF trace messages in the bottom half of the screen. I believe it’s quite straightforward what you can do here: pause, resume, clear and filter the messages. A maximum of 1000 messages are saved in the trace queue. When this size has been reached the message at the top of the queue is removed in favour of the most recent message. Check out the column headers of the trace messages, they are easy to understand.

Web farms in .NET and IIS part 5: Session state management

Managing session state is vital in a web farm environment. Many websites simply cannot function without maintaining state throughout the user’s visit. Session variables are maintained by the web server and a cookie is saved to the client. This cookie will inform the server who the user is and the server can then extract the session information from it. You will need to give extra consideration to state management in a web farm scenario.

Imagine that the web request is routed to farm machine A and the code invoked by the request relies on some parameters stored in the session. If the same user is then routed to machine B in the next web request which also relies on the session state then the results may unpredictable: as the value is not available in the session then a null pointer exception may be thrown, or if you put in a guard clause then that variable may be assigned some default value thereby “forgetting” what the value was in the previous web request. And then the wrong default value may be stored in the database instead of the one the user has selected.

In this post we discussed session stickiness which makes sure that the same user is directed to the same machine on subsequent requests. We also said that we should avoid this solution so that the load balancer can pick the “right” machine in the farm based on the actual load. In case you need to deploy a Classic ASP site in a web farm then this is your only option: Classic ASP doesn’t have any built-in session state solution that works with a web farm.

You can define session state handling in IIS. Let’s see what the options are.

Session state management

Every web developer must have used the session state in some form. A classic example is when a user fills in a step-by-step signup questionnaire then the values are store in a session:

Session["FirstName"] = txtFirstName.Text;
Session["LastName"] = txtLastName.Text;

The question is how and where these session states are stored.

In IIS you can open the session state manager by choosing the web site and then clicking the Session State icon:

Session state icon in IIS

This opens the following window:

Session state options in IIS

The default choice is in-process. This means that the session state data is stored inside of the worker process called w3wp.exe. You can think of this as storing the session inside the application pool. This option provides the fastest retrieval of the session state as there’s no need to connect to another computer or database to read or store the session data. However, there are several disadvantages which are also related but not limited to web farms:

  • Session management runs inside the worker process which means that it’s consuming memory in the worker process. This memory could be used by other functionality of your web site.
  • The session state is lost as soon as the application pool of your website in IIS is recycled. It is gone forever.
  • This option is particularly dangerous in a web farm environment. Take the example I mentioned above: one farm machine stores the session state but not the other. Subsequent web requests from the same user may not read the correct session state

This last point forces us to introduce sticky sessions which beats the purpose of load balancing so you should always try to avoid this path.

The points above will certainly lead to bad user experience in the form of inexplicable exceptions, unexpected values and irritating bugs. Each machine in the web farm will store its own little subset of the full session state:

In process session management

Salvation comes in the exact opposite of in-process state management: out-of-process state management. This option simply means that instead of storing the session state within the application pool we store it somewhere else, in a place which is independent of the website:

  • ASP.NET session state service: this provides a somewhat slower service than the in-process variant as we need to make calls to a remote server. All session data is stored in memory so shutting down the state machine will wipe out all session data as well
  • SQL server, where we store the state in a database: this option provides the most reliable service but is also the slowest as all data must be read to and from a database. Session data is not lost even if the state machine is shut down
  • A third party component, possibly commercial or self-built

So we move out the session state from each individual farm machine to one central place where the session can be read and stored:

Out of process state management

ASP.NET state service

Open the Services management window and you should find the following service there:

ASP.NET state service in Services window

So it’s a plain old Windows service. Every machine that has a .NET framework installed has this service. As you see by default it is turned off and is set to a manual start mode. Change this to automatic and start the service to activate it:

Set service to automatic

Normally you would do this on the designated state service machine. This will allow external machines to connect to the state service and put session data in and out of it and it’s all stored in memory. Restarting either the state server or just that Windows service will erase the session data.

Also, the service cannot be accessed remotely by default. To enable that option you need to set the value of the following registry key to 1: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aspnet_state\Parameters\AllowRemoteConnection. Don’t forget to restart the state service after the registry key change.

So this Windows service is easy to use but there’s a significant disadvantage: the State Server does not provide any failover option. If the service is stopped or the machine running the state service is shut down then your session variables will not be available and cannot be saved. If you are willing to take that risk then you’ll be good to go in a couple of minutes.

SQL server

As mentioned above this provides the most reliable session service but it also the slowest. If you rely on the session state a lot in your code then there will be a lot of read and write operations to and from your SQL server affecting its performance. However, the added redundancy may well compensate for the slight increase in the performance overhead.

You can have a cluster of SQL Servers configured with SQL AlwaysOn so that even if one state machine goes down then the other machine can take over. As state is persisted in database tables it is only lost if the table data is wiped out. Otherwise it is available even if the machine is restarted.

You must prepare your database to fit the session state schema. This can be done using the aspnet_regsql.exe tool which is located in the following folder:

ASP net reg sql tool location

You can only perform the necessary schema change from the command line. If you double-click the executable file then a GUI will open but that GUI is used for setting up ASP.NET membership, profiles and role management tables in the database. It is not meant for state management purposes.

Make sure that the user you provide in the command line arguments has admin rights to the database otherwise the command will fail. The command will need access to msdb system-stored procedures.

The command line arguments include a paramenter called -sstype which stands for the SQL Server session type. It has 3 options: ‘t’ for temporary, ‘p’ for permanent and ‘c’ for custom. T and p create a database called ASPState. T saves the session in the tempdb database so this data is only temporary: session state is lost if SQL Server is restarted. P on the other hand causes the session data to be saved in the ASPState database and allows the session state management to use the same SQL server.

Make sure to open port 1433 on the machine where you’re intending to save the session data. Note that the parameters are case-sensitive. Examples:

aspnet_regsql.exe -ssadd -sstype c -S sql.mycompany.local -d SQLState -U adminUserName -P adminPassword

-ssadd simply means to add SQL Server session state. The server is specified though -S, -d specifies the database, -U the username and -P the password.

aspnet_regsql.exe -ssadd -sstype c -S sql.mycompany.local -d SQLState -E

This is the same as above except that it uses Windows authentication.

aspnet_regsql.exe -ssadd -sstype p -S sql.mycompany.local -E

Here we ask the tool to create the ASPState database, which is the default name of the session state database, hence we don’t need the -d parameter.

You can view all parameters by typing aspnet_regsql.exe /?

Third party tools

The following tools are all reliable session managers that are built for highly available, highly scalable web farms. They either maintain session data in-process and replicate all changes immediately or store the session data out-of-process through a custom worker process that resides on each server.

Preparing your web site

You need to declare in web.config in case you want to use an out of process state management option. Create a new MVC internet application or simply open the one we used in the previous blog post. Open web.config. You need to declare the session management options within the system.web node. Example:

<sessionState mode="StateServer"								stateConnectionString="tcpip=machinename:42424">

If you want to use the StateServer service on the local machine then you can ignore the stateConnectionString attribute.

Use intellisense to check the attribute options for the session state node. You will find the SQL variant there as well. You’ll probably recognise the purpose of the mode attribute and its value. The state connection string will be the address of the state machine and the port of the ASP.NET session service. By default it is listening on port 42424.

In case you’re opting for the SQL Server type of solution then this section may look like the following:

<sessionState mode="SQLServer" allowCustomSqlDatabase="true"
	sqlConnectionString="the connection string name to the server">

Where sqlConnectionString refers to the name of the connection string in the connectionStrings section of web.config which holds the details of the state table. The connection string itself may take up the following format:

<add name="ASPStateConnectionString" connectionString="Data Source=[DB machine name where state DB is stored];Initial Catalog=ASPState;User ID=[db admin user name];Password=[db admin password]"
   providerName="System.Data.SqlClient" />

We’re not done yet, we also need to add a machine and a validation key within the system.web node, that may look something like this:

	validationKey="some long hashed value"
	decryptionKey="another long hashed value"

The validation key is used to validate encrypted data such as the ViewState to ensure it has not been tampered with. The decryption key is used to encrypt and decrypt forms authentication data and viewstate when validation is set to TripleDES.

The default setting for the validation key is AutoGenerate which does exactly what the name applies: the key will be generated automatically by IIS. The default generation mode for the decryption key is IsolateApps. It generates a unique key for each application by using the application ID.

We need this common machine key as we want to run this on several machines so we don’t want the machine key to be automatically generated. The default IsolateApps mode for the decryption key is OK to use in a web farm scenario but the validation key must be common across all servers so we want to produce it ourselves. There are ASP.NET machine key generators out there on the net, you can search yourself, but here are some:

Easiest way to generate MachineKey
Generate ASP.NET Machine Keys
ASP.NET resources

It’s important to emphasise that we want to have static machine key values. The opposite scenario would be that each web farm machine has its own machine key, which will have different values of course and then they access different versions of the session state service.

If you prepare web.config and then deploy your web site using the deployment profile we built in the previous post then the values in the Session State window in IIS will be populated with the values defined in web.config:

State server options filled out in IIS after deploy

Check the machine key in IIS as well:

Machine key icon in IIS

You’ll see that even the validation and decryption keys have also been filled out:

Machine key filled out in IIS

Now that we have set up the out of process state management we can recycle the application pool as much as we like the session state variables will be preserved. If for whatever reason you want to destroy the session state you have the restart the ASP.NET State Service in the Services window or restart the state machine itself.

If you recall from the previous post we set up a batch file that deploys the entire IIS configuration from the staging server to the web farm servers. That will ensure that even the machine keys will be “exported” so that we have one common machine key for the entire farm.

In the next post we’ll start looking at Web Farm Framework.

Web farms in .NET and IIS part 4: Code deployment

So now we have our web farm in place where the servers in the cluster are identical machines: they have the same CPU, same RAM, same OS etc. This is an important point as you certainly want to provide the same user experience regardless of which web farm server receives the web request.

A good way to ensure this is through using templates in a virtualisation environment, such as VMWare. You define a template for a model machine with all the necessary physical traits such as the CPU, disk size, RAM etc. You then use this template to create new virtual machines. This process is a lot smoother and cheaper then buying new physical machines.

The next step is code deployment. You typically want to deploy your website on all machines in the farm, right? You can of course manually copy over your deployment package to each and every server one by one but that’s inefficient and tedious. A staging server can help you out.

Staging server

A staging server is not part of the web farm therefore it won’t get any web requests. However, it will also have IIS installed just like the machines in the web farm. We can first deploy our code to that IIS and we can use that IIS to deploy the website to the web farm servers. The staging server can act as an environment where we check if our code is working properly, if the configuration is working fine etc. It is the last stage before actual employment so ideally the staging server has the same physical properties – CPU, RAM, etc. – as the servers in the web farm:

Staging server

There’s a free tool from Microsoft that you can use to deploy your code: MSDeploy, or Web Deploy. MSDeploy is a command line tool that Visual Studio uses behind the scenes if you choose that deployment method when publishing your code. It can be used to migrate IIS configuration, web content, Windows components. It is a powerful tool which is well suited for web farms because it allows to synchronise some more difficult items such as the registry or the GAC.


Open Visual Studio 2010 or 2012 and create a new MVC internet application. I will go through the steps using VS 2012 but web deployment is available in VS 2010 as well. You can leave the default website content as it is, we’re only interested in the deployment options. You can open the publish options from the Build menu:

Publish web menu point

This opens the Public Web window:

Publish Web window

In the Select or import a publish profile drop down list select New… Give some meaningful name to the deployment profile:

Give name to deployment profile

You can then select the deployment method:

Deployment method option selector

The Web Deploy publish method represents the MSDeploy tool mentioned above. Like in parts 2 and 3 of this series I don’t have access to any private web farm so I’ll just deploy this web site to the IIS instance of my own computer.

UPDATE: in this post I show how to set up Web Deploy with a real external web server.

Open the IIS manager and add a new web site:

Add application IIS

Give it some name and select the physical location of the deployment package. You can create a new folder using the Make New Folder button.

Back in VS we can now specify the server as localhost and define the application name which we’ve just set up:

Web deploy options

You can validate the connection using the Validate Connection button which requires that you start VS as an administrator.

The Service URL can be the location of your staging server or that of a hosted environment that you’re deploying your site to:, or whatever URL you gave that IIS website. Press Publish and you may be greeted with the following error message in VS:

Web deployment failed due to wrong runtime version

Back in IIS we need to modify the default application pool that IIS created automatically for the new web site:

Default application pool in IIS

Right-click that application pool and select Advanced settings from the context menu. Change two things in the Advanced Settings window:

Modify application pool in IIS

Run the deployment from VS again and it should proceed without errors:

Web deploy success

You can even check the contents of the deployment folder you specified in IIS to verify that the web site was in fact deployed:

Deployment package in folder

You can even see the same structure in IIS under the website:

Web site contents in IIS

In a real life environment we would check this code into source control, such as GIT or SVN. The staging server would pull out of source control the latest check-in code but for now we’ll just pretend that this localhost environment represents our source control server. So we’ll push this code to our staging server thereby doing what a source control would do for us automatically.

The msdeploy tool is normally located at C:/Program Files/IIS/Microsoft Web Deploy V3. At the time of writing this post V3 was the latest available version. By the time you read this there may be a newer version of course. Open a command prompt, navigate to that directory and run msdeploy.exe without any arguments. You should get a long list of options that msdeploy can be used with:

MSDeploy options in command window

We can write a short batch file to carry out the necessary msdeploy commands for our deployment purposes. In the batch file, which we can call deploy.bat we first specify the directory:

cd “\Program Files\IIS\Microsoft Web Deploy V3”

Then we can fill up the file with msdeploy commands. To deploy from the current location, i.e. localhost, to our staging environment, we can write like this:

msdeploy -verb:sync,computerName=[name of staging server machine]

We’re instructing msdeploy to pull the code out of on the local machine and deploy it to on the staging server. This command will simply take the contents of the website and push it over to the given destination.

Then from there it’s equally straightforward to deploy to our web farm machines:

msdeploy -verb:sync -source:webServer,computerName=[name of staging server machine] -dest:webServer,computerName=[name of first machine in web farm]
msdeploy -verb:sync -source:webServer,computerName=[name of staging server machine] -dest:webServer,computerName=[name of second machine in web farm]

So here we don’t just copy the application itself but the entire IIS config and its contents from the staging server and push it over to the web servers one by one. We’re taking the staging server and push it out to farm machine 1 and 2.

We do this because we want to make sure that the web farm machines have the exact same IIS configuration as the staging one. Remember that we can run tests on the staging server where we fine-tune some IIS settings, like response headers or whatever, and we want those settings to be copied over to all live web servers. We want this to ensure that every member in the farm behaves the same, i.e. the end users get the same behaviour. If we only copy the application contents from the staging machine to the web farm machines then it’s not guaranteed that the IIS settings across the web farm will be the same as on the staging machine.

In other words the staging server is a central place in the system. It is a blueprint for all live web servers in the web farm: the code, the configurations, the IIS settings etc. available on the staging server are the ones that take precedence. It represents the most recent stage of the deployment environment. You shouldn’t go out and change the settings on the web farm servers and hope for the best. Instead, you “play” with the staging server settings and when you’re satisfied with the results you can push out the entire IIS config and its contents to the web farm.

If you want to get really advanced then you can employ a PowerShell which pulls out the web server from the farm just before deploying the code to it to make sure that this machine doesn’t get any traffic during the deployment process. The script would do this to every machine in the farm one by one.

Other content replication tools

MSDeploy is by no means the only tool to replicate content. You can check out the following alternatives:

  • Distributed File System (DFS)
  • Robocopy
  • Offline folders
  • Web Farm Framework

DFS is a solution to keep servers in sync. It consists of two parts: DFS Namespaces (DFSN) and DFS Replication (DFSR). They make up a powerful tool to achieve high availability and redundancy. DFSN ensures that if one server fails then another takes over without any configuration changes necessary. DFSR on the other hand gives you the ability to replicate content over either a LAN or a WAN.

Robocopy – Robust File Copy for Windows – is a command line tool by Microsoft that will replicate content between servers, either one-way or two-way. Its main advantage is the easy-to-configure replication between folders. However, it doesn’t offer a method to redirect to the backup server is the primary server fails. With DFS this is possible to achieve.

The next option, i.e. offline folders, is rather a tool to enable the usage of offline folders which offers faster performance when the network is slow and offline files when the network is completely down. The result is that configuration files will always be available even in the case of network problems. This technique is known as Client Side Caching (CSC): in case of a network failure IIS will use the cached version until the network connection is restored.

We mentioned the Web Farm Framework in the first part of this series: it’s an advanced plug-in to IIS made by Microsoft which can fully automate the content replication tasks. We’ll take a closer look at WFF in an upcoming post.

Web farms in .NET and IIS part 3: Application Request Routing ARR


In this post we’ll look at the basics of Application Request Routing. The target audience is beginners who have not worked with ARR before and want to get going with it.

As mentioned in the first post of this series ARR is an extension that can be added to IIS 7.0 and above.

ARR has a lot more available functions than Network Load Balancing (NLB) making it the better load balancing option. However, ARR doesn’t have its own solution for high availability so it cannot handle failures to the server hosting ARR – the result is that the ARR server becomes a single point of failure. This is where NLB enters the scene as it provides high availability – ARR and NLB together make up a very good team.

ARR comes with the following advantages compared to full-blown hardware based load balancers:

  • Cost: if you have IIS7.0 or above you can install ARR free of charge
  • Ease of use: this is of course relative, but if you have some basic knowledge of IIS then the learning curve will be minimal
  • Performance: ARR can handle very large sites with ease – the first resource limit that ARR runs into is network. Most networks support 1 Gbps or more, so this is usually not an issue
  • Flexibility: ARR offers load balancing based on any server variable, URL, cookie etc.

ARR has some disadvantages:

  • As mentioned above ARR doesn’t have its own built-in solution for high availability
  • It does not offer the same range of features as more complete hardware based products. E.g. it lacks SEO treatment and DDoS handling. Some of these shortcomings can be solved with other products, such as NLB or Request Filtering

You can get pretty far with NLB and ARR unless you have some very specialised needs that only expensive commercial products can provide.

ARR is a software based reverse proxy solution – for an explanation of these terms check out the first post in this series. It supports all common load-balancing algorithms, such as server weighting, round-robin etc. The following general list shows the features available in ARR:

  • Health checking
  • Caching
  • Can work as a Content Delivery Network (CDN)
  • SSL offloading
  • HTTP compression
  • URL rewrite
  • Usage reporting
  • Sticky sessions
  • Programming and automation support

ARR only handles HTTP traffic, so it cannot work in conjunction with FTP, RDP etc.


ARR routing goes through three touchpoints:

  1. IIS site binding: your website will of course need to have a valid URL which we can call the ARR base. It is a site that appears below the Sites node in the IIS manager
  2. URL rewrite rule: this happens at the IIS level
  3. The server farm

When a web request hits the ARR server it is first caught by the IIS binding. A standard binding needs to exist on a website. This is the same process as setting up a new website in IIS under the Sites node. It can be a generic website with no real content – it is only used for the HTTP and HTTPS bindings. Also, it is here where SSL offloading occurs.

The request is then caught by URL Rewrite as long as an appropriate rule exists. This happens at the PreBeginRequest step in the IIS pipeline so it is performed before any other site functionality. URL Rewrite rules can edit server variables as well.

In the last step the server farm gets the request from URL Rewrite. It determines which server to send the request to based on the load balancing algorithm and client affinity settings.


You can download ARR on your designated routing server using the web platform installer available here. Search for Application Request Routing in the search window of the installer and select the below version:

ARR in web platform installer

By the time you read this post there may of course be a new version of the extension.

Click Add and Install in the installer window. The installer will fetch all the dependencies as well.

As in the case of Network Load Balancer I don’t have a private network of Windows servers. Therefore I can show you the steps you need to take in order to create a web farm but will not actually create one in the process.

As soon as you install ARR you will see a new node called Server Farms in the IIS manager:

Server farms node in IIS

Right click on that node and select Create Server Farm. The Create Server Farm window will appear where you can add the Server farm name:

Create server farm window

You can provide any name here, like in the case of the, but give it some URL-type of name, such as It’s a good idea to provide the same name as the URL for the ARR version of your website. Example: if your website is then the ARR version may well be Click Next to go to the Add Server window:

Add Server window in ARR setup

Here you can add the servers that are the members of the farm. Add the server address one by one in the Server Address textbox and click Add after each entry. The servers you will appear in the list box. Click Finish when you’re done with this step. ARR will then ask you if you want to create any URL rewrite rules:

Ask if URL rewrite rules should be added

Click yes. You’ll see that the server farm will appear in the Server Farms list of the IIS manager. Open the Server Farms node, select the name of the farm you’ve just created and you will see some additional setup options:

Server farm setup options in IIS

The default options are sufficient to test the network. Open a web browser and navigate to the ARR name of your website such as The request should be routed to one of the web farm members you specified during the setup.

Let’s look at some of the options in the IIS manager. Clicking on Caching will open the following window:

Caching options window ARR

The default setting is that anything that goes through ARR will be cached for 60 seconds and disk cache is enabled. So if 2 web requests ask for the same resource within a minute ARR will hand back the cached version to the second request without consulting the web tier. You can even set whether the caching mechanism should cache the results by the query string or not. Example: and will probably return different results from your backend database so it’s a good idea to cache by the query string.

So what is cached? ARR follows the caching rules set out in RFC2616. By default all static content such as css, js etc. is cached.

If your site streams media then checking the Enable request consolidation checkbox can be a good idea. It will consolidate all the streaming requests to reduce the number of requests.

Also, it is recommended to turn off the idle time setting of the application pool:

Idle time out in IIS

By default this value is set to 20 minutes. This means that the application pool will enter “sleep” mode if it sits idle, i.e. receives no web request for 20 minutes. Set that value to 0 to turn off this feature, which is recommended anyway, so not only in web farm scenarios. This keeps your site actively running while maintaining the health checking even during quiet times.

ARR can even function as a Content Delivery Network to front entire web farms. Check out the resources on this page for a video user guide.

The Health Test window looks as follows:

Health test window ARR

Here you can schedule explicit health checks and/or rely on live traffic tests. Live traffic testing watches for errors with the live traffic. If it sees what you define as a failure it marks that server as unhealthy. The main advantage of this method is that is watches for errors with any type of page request not just a single testing URL. The main disadvantage is that if the developers release a bad, untested version of the website then eventually all web farm machines may be turned off as all of them may eventually produce the same error if the same page is requested by the clients. This makes a DoS attack a breeze and even completely innocent clients can break down your site if they hit F5 often enough. Also, if a live traffic test shuts off a web server then that server is not brought back online automatically. Therefore it’s very important that you set up explicit URL testing as well and don’t rely on live tests exclusively.

You can configure live traffic tests as follows:

  • A comma separated list of failure codes that will cause the web server to be marked as unhealthy. You can even define a range with a hyphen, e.g. 400-500, or mix the styles such as 500, 510-599
  • The number of maximum failures that must occur during the failover period before the server is marked unhealthy
  • The failover period in seconds is used along with the maximum failures to determine if there are too many failures during the failover period. If this value is set to 0 then live traffic testing is disabled.

With explicit URL testing you ask ARR to send a request to the web farm members with specific time intervals and inspect the response. You can provide the URL in the URL text box. The test request will be sent to each machine in the farm. It’s highly recommended that you set up explicit URL checks so that unhealthy servers are not allocated any web requests. You can enable explicit URL tests by providing a URL in the appropriate text box. We’ll come back to this type of test a little later, but here are the parameters you can set:

  • The test URL whose response is inspected by ARR
  • The time between tests in seconds – if the test fails ARR will continue to send the same request and bring the server back online if the test passes
  • The timeout in seconds before the health test gives up on a page that takes too long. If ARR receives no response before the timeout then the server is marked as unhealthy.
  • The Acceptable status codes field works the same way as failure codes in the case of live traffic tests
  • You can also perform a content match to ensure that a certain word or phrase can be found on the page.

Bear in mind that ARR will mark the server as unhealthy after the very first bad response. Therefore make sure that the testing interval is not too long as even fully functioning and healthy servers may produce bad responses from time to time.

In the bottom of the same window you’ll see another text box where you can define the number of minimum servers. The idea here is that if you know that you need a certain number of healthy servers to reasonably handle the load then the health test shouldn’t allow the server farm to drop below that level. Example: your web farm has 10 machines and you need a minimum of 6 to handle the load on an average day. If the number of healthy servers drops to 5 then it may be a better idea to bring all machines back online again: 5 out of 10 users will receive good experience, so not all visitors will be impacted. The quality level is difficult to predict: your users may see intermittent errors that only occur if their request is directed to one of the bad servers. This is a tradeoff: the web farm will not be rendered completely useless due to overloading but the user experience may not be top class either. If this limit is reached then ARR will ignore all scheduled tests and bring back all servers online again.

The test URL should be a very simple self-contained page: it should not access the database or your application logic. If your database or application layer fails then all the web farm machines will be taken out of action. The ideal test page contains only some static HTML. Make sure that the test page only fails if there’s a problem with the specific server the page is employed on. Otherwise, if the test page fails on every server then the entire web farm will be useless as all servers are brought offline.

You can pick the load balance algorithm in the Load Balance window, meaning how would like to set the rules for web request routing:

Load balance algorithm options ARR

The default option is Least current request which is probably the most logical choice. This option means that the web request will be routed to the web farm member that has the least amount of request. Weighted round robin tells ARR to route the requests to each web farm member in turn: first to server 1, then 2 then 3, back to 1, then 2, then 3 etc. regardless of the current load each machine. You can find the explanation of each algorithm in the first post of this series.

If you change the default options you’ll need to select Apply to the right:

Apply changes to ARR in IIS

The Monitoring and Management window will show you the web request statistics and current health state of each member of the web farm:

Monitoring and management ARR in IIS

You can even get some real time disk cache statistics in the bottom of this window:

Real time disk cache stats ARR

Proxy allows us to configure how the packets are forwarded:

Proxy settings ARR

Preserving the X-Forwarded-For header can be useful if you want to see which IP the client’s request originated from.

In the Routing Rules window we can set various advanced features.

Set routing rules ARR

You can e.g. let the ARR server take care of SSL encryption and decryption by selecting the Enable SSL offloading option which is set by default. Keep in mind that ARR is a reverse proxy load balancer meaning that the web farm servers will see the http request coming from ARR and not from the client. If ARR is responsible to encrypt and decrypt SSL then the web farm will get HTTP requests from ARR even if the original request was sent through HTTPS. If it is critical in your environment that the web farm receives HTTPS requests then a tool called ARR Helper can be something to consider: it digs out the details of the original HTTPS message from the ARR server and writes them back to the original locations thereby fooling the web servers into thinking that the request came from the client in the form of HTTPS. The tool can be downloaded from this link. It comes in two versions: 32bit and 64bit. The tool must be installed on the web servers, not on the ARR machine.

In the Server Affinity window you can set whether you want sticky sessions or not.

Server affinity window in ARR

The default setting is that there are no sticky sessions. Selecting Client affinity means that the same user will be directed to the same machine during the web session. Session stickiness will be cookie based, you can even set the name of the cookie in the Cookie name text box.

With the web farm and the load balancer in place you can test how ARR behaves if one of the web farm servers is taken out of action. Go to the IIS manager of a server and stop the application pool of the website deployed there:

Stop application pool ARR web farm

If you selected the round robin routing algorithm then you will see the following: a normal 200 response from the web server still up and running followed by a 503 from the one you’ve just shut down, then back to healthy server, then again to the inactive one etc. We can set up a status check to avoid this behaviour so that ARR can determine if a web server is healthy before routing any web requests to it.

Go back to ARR and select the Health checks option. Enter a URL which ARR can send requests to. It can be a page dedicated to the health checks, a very simple page with no value to a “real” visitor. Its only reason to exist is to check the state of the web server. How you implement such a logic is up to you, e.g. check if the page can connect to a web service. The outcome of that logic should be a simple boolean value: is this web server in a healthy state or not.

Then you can set the interval for the health checks in seconds:

Set interval of scheduled health checks

Note that these checks are real web requests so it creates some additional load on your web site. Therefore make the status URL as lightweight as possible.

Then you can set the upper limit of the response timeout and the acceptable status codes as well. The response match text box has the following purpose: say that your status page writes a specific string in the response if the status is fine, such as “True” or “Success”. You can put that string in this text box. If the response from the status page is different from this string then the machine is deemed unhealthy.

You can test the status response by pressing the Verify URL test button:

Verify URL test button ARR

If you again turn off the application pool of one of the farm members then you should see that ARR does not route any traffic to that server. Also, the Monitoring and Management window should tell you that the server is not healthy:

Unhealthy server in web farm

High availability for ARR

As mentioned before ARR doesn’t have any built-in high availability feature. If the ARR server is down then the web farm will be unavailable. This doesn’t sound too convincing as one of the reasons we set up web farms is to ensure high availability, right? One solution is to go for a full-blown hardware based commercial load balancer but those typically cost a lot of money. If you are looking for a cheaper alternative then Network Load Balancing (NLB) can be useful. NLB is a load balancer with limited capabilities but can be used together with ARR. NLB must be installed and configured on the web servers of the farm. See the previous post on NLB.

This concludes our discussion on the basics of ARR.


A great site

iReadable { }

.NET Tips & Tricks

Robin Sedlaczek's Blog

Love to use Microsoft Technologies

HarsH ReaLiTy

A Good Blog is Hard to Find

Ricos Blog zu Softwaredesign- und architektur

Ideen und Gedanken rund um Softwaredesign und -architektur, Domain-Driven Design, C# und Windows Azure

the software architecture

thoughts, ideas, diagrams,enterprise code, design pattern , solution designs

Technology Talks

on Microsoft technologies, Web, Android and others

Software Engineering

Web development

Disparate Opinions

Various tidbits

chsakell's Blog

Anything around ASP.NET MVC,WEB API, WCF, Entity Framework & AngularJS

Cyber Matters

Bite-size insight on Cyber Security for the not too technical.

Guru N Guns's

OneSolution To dOTnET.

Johnny Zraiby

Measuring programming progress by lines of code is like measuring aircraft building progress by weight.

%d bloggers like this: