Introduction to OAuth2: Json Web Tokens

Introduction

JSON web tokens are a sort of security token. As such, it is used for authentication purposes, and has similar attributes like the XLM-formatted SAML tokens we met in the series on Claims Bases Authentication. It shows the issuer of the token, the claims about the user, it must be signed to make it tamper-proof and it can have an expiration date. If you then log on to a web page then the authentication server will send back a security token that contains the data mentioned above. Upon successful authentication the web site will consume the token.

Again, like in the case of the SAML tokens there must be a trust relationship between the consumer and the issuer of the token. This ensures that even the contents of the token are trusted. The consumer knows about the key that the issuer uses to sign the token. That key can be used for validation purposes. At the time of writing this post SAML tokens are the most commonly used security tokens. SAML has quite a complex structure with a strict schema involved. It is very expressive with various encryption and signature options.

This last point is actually a limitation in the world of mobile devices. It’s needless to say how widespread mobile devices are in the world. There are probably more tablets and smartphones available nowadays than laptops and stationary computers. It feels like it’s only programmers like me who need computers… Mobile devices must perform their tasks with resources that are limited in comparison to computers. They are not well suited for parsing complex XML tokens.

This is where the JSON web tokens enter the scene with their simplified structure. The data is JSON which is more compact than XML and even mobile devices have the ability to parse them. JSON is also native to JavaScript which has grown into a very important language with applications such as Node.js, Windows 8 store apps, MongoDb etc. There are still a number of symmetric and asymmetric encryption options available for the token content and signature but it’s limited compared to what the SAML standard has to offer. However, the most widely accepted algorithms, such as RSA and AES are available and they will suit the vast majority of needs.

With the fast advance of mobile devices JSON web tokens will soon become – or already have? – the new standard for security tokens. They are already the de facto standard for OAuth2 and OpenID Connect in the modern mobile implementations.

Structure

A JSON security token consists of two parts:

  • Header with some metadata, e.g. on the algorithms and keys used to encrypt and sign the message
  • Claims

If you don’t know what claims are then make sure you understand the basics: start here.

These claims can be token specific, such as Issuer, Expiration date, or they can be defined by the application. You as the programmer are free to define what goes into the application-specific claims, such as FirstName, Email, NameOfPet, FavouriteColour, you name it.

If you’ve read through the series on Claims Based Auth available on this blog then the token specific claims are probably familiar to you:

  • Issuer: the identifier of the issuer of the token so that the recipient knows where the token is coming from
  • Audience: shows who the token is targeted at so that the recipient doesn’t use a token meant for a different application or website
  • IssuedAt: when the token was issued
  • Expiration: the expiration date of the token
  • Subject: typically a user ID that describes who the token is meant for

To keep the token size limited these claim types are abbreviated:

  • Issuer: iss
  • Audience: aud
  • IssuedAt: iat
  • Expiration: exp
  • Subject: sub

This is what a header can look like:

{
"typ": "JWT"
, "alg": "HS256"
}

Type: JWT means of course Json Web Token. Alg shows that the HMACSHA256 algorithm was used to sign the token.

The claims section can look like this:

{
"iss": "issuer name, usually some URL http://www.myauthenticationserver.com",
"exp": "a date in UNIX date format i.e. the number of seconds since 1970/01/01",
"aud": "the URL of the consumer likehttp: //www.mygreatsite.com",
"sub": "user identifier such as bobthebuilder"
}

Other claim types can be useful in the claims section: e.g. “client” to identify the application that requested the token. or “scope” to show the list of allowed operations for authorisation purposes:

{
"scope": ["read", "search"]
}

The list of claims can be extended as you wish, just as we saw in the case of SAML claims based authentication.

To make the token even more compact, the different sections are base64 encoded:

sdfgsdfgdfg.hrtg34twefwf4fg5g45gg.wsefefg345e4gf5g

These are just random characters, but locate the 2 periods in this string. They are delimiters for:

  1. Header
  2. Claims
  3. Signature

…where each section is individually base64 encoded.

Some code

Open Visual Studio 2012 or higher, create a console application and add the below package from NuGet:

JWT package from NuGet

In addition, add a reference to the System.IdentityModel library.

There’s a number of ways to exchange JWT tokens between a sender and a receiver. In the below example I’ll use an RSACryptoServiceProvider to sign the JWT so that the receiver can validate it. If you don’t what RSA and asymmetric encryption mean then make sure to read upon it in the blog post mentioned above.

In short: the sender, i.e. the issuer of the token will have a pair of asymmetric encryption keys. The public key can be distributed to the receivers. The receiver will use the public key to validate the signature of the JWT token.

We won’t build a separate sender and receiver, that’s not the point here, but we want to simulate that the sender has access to both the private and public keys and the receiver only has the public key.

The following method will construct a valid RSA key pair:

private static RsaKeyGenerationResult GenerateRsaKeys()
{
	RSACryptoServiceProvider myRSA = new RSACryptoServiceProvider(2048);
	RSAParameters publicKey = myRSA.ExportParameters(true);
	RsaKeyGenerationResult result = new RsaKeyGenerationResult();
	result.PublicAndPrivateKey = myRSA.ToXmlString(true);
	result.PublicKeyOnly = myRSA.ToXmlString(false);
	return result;
}

…where RsaKeyGenerationResult is a DTO:

public class RsaKeyGenerationResult
{
	public string PublicKeyOnly { get; set; }
	public string PublicAndPrivateKey { get; set; }
}

In GenerateRsaKeys() we generate an RSA key and save its full set of keys and the public key only in two separate parameters of the RsaKeyGenerationResult object.

This is how to build and serialise the token:

RSACryptoServiceProvider publicAndPrivate = new RSACryptoServiceProvider();			
RsaKeyGenerationResult keyGenerationResult = GenerateRsaKeys();

publicAndPrivate.FromXmlString(keyGenerationResult.PublicAndPrivateKey);
JwtSecurityToken jwtToken = new JwtSecurityToken
	(issuer: "http://issuer.com", audience: "http://mysite.com"
	, claims: new List<Claim>() { new Claim(ClaimTypes.Name, "Andras Nemes") }
	, lifetime: new Lifetime(DateTime.UtcNow, DateTime.UtcNow.AddHours(1))
	, signingCredentials: new SigningCredentials(new RsaSecurityKey(publicAndPrivate)
		, SecurityAlgorithms.RsaSha256Signature, SecurityAlgorithms.Sha256Digest));

JwtSecurityTokenHandler tokenHandler = new JwtSecurityTokenHandler();
string tokenString = tokenHandler.WriteToken(jwtToken);

Console.WriteLine("Token string: {0}", tokenString);

You’ll recognise the parameters in the JwtSecurityToken constructor. You’ll see that we used the full RSA key to sign the token. The SecurityAlgorithms string enumeration stores the fully qualified names of the signature and digest algorithm. Those are compulsory arguments in the constructor.

Run the code up to this point and you should see a long string with the 3 segments mentioned above.

If your organisation has a valid X509 certificate then it can be used in place of the RsaSecurityKey object. Check out the X509AsymmetricSecurityKey object which accepts a 509Certificate2 object in its constructor. You’ll need to export the certificate with both the private and public keys and save it as a .pfx file. You can do that using the certmgr snap-in in Windows. Use the full file name of the .pfx file in the constructor of the X509Certificate2 constructor.

You can transmit the token in a number of different ways: in the header of an HTTP request, in the query string, in a cookie, etc. It’s now a string that you can send to the consumer.

The consumer can validate and read the claims from the token as follows:

JwtSecurityToken tokenReceived = new JwtSecurityToken(tokenString);

RSACryptoServiceProvider publicOnly = new RSACryptoServiceProvider();
publicOnly.FromXmlString(keyGenerationResult.PublicKeyOnly);
TokenValidationParameters validationParameters = new TokenValidationParameters()			
{
	ValidIssuer = "http://issuer.com"
	,AllowedAudience = "http://mysite.com"
	, SigningToken = new RsaSecurityToken(publicOnly)
};
			
JwtSecurityTokenHandler recipientTokenHandler = new JwtSecurityTokenHandler();
ClaimsPrincipal claimsPrincipal = recipientTokenHandler.ValidateToken(tokenReceived, validationParameters);

The client receives the base64 string which can be used in the constructor of the JwtSecurityToken object. We then use the public-key-only version of the RSACryptoServiceProvider to simulate that the receiver only has access to the public key of the sender. The TokenValidationParameters object can be used to build the validation logic. The JwtSecurityTokenHandler object is then used to validate the token. Upon successful validation you’ll get the ClaimsPrincipal which you’ll recognise from the posts on claims based auth mentioned above.

Set a breakpoint after the last line of code and inspect the contents of the claimsPrincipal object by hovering over it in VS.

The encoded JWT token can be decoded on this web page:

Jwt decoder page

Read the next part in this series here.

You can view the list of posts on Security and Cryptography here.

Getting a return value from a Task with C#

Sometimes you want to get a return value from a Task as opposed to letting it run in the background and forgetting about it. You’ll need to specify the return type as a type parameter to the Task object: a Task of T.

.NET 4.0

Without specifying an input parameter:

Task<int> task = new Task<int>(() => 
{
    int total = 0;
    for (int i = 0; i < 500; i++)
    {
        total += i;
    }
    return total;
});

task.Start();
int result = Convert.ToInt32(task.Result);

We count to 500 and return the sum. The return value of the Task can be retrieved using the Result property which can be converted to the desired type.

You can provide an input parameter as well:

Task<int> task = new Task<int>(obj => 
{
    int total = 0;
    int max = (int)obj;
    for (int i = 0; i < max; i++)
    {
        total += i;
    }
    return total;
}, 300);

task.Start();
int result = Convert.ToInt32(task.Result);

We specify that we want to count to 300.

.NET 4.5

The recommended way in .NET 4.5 is to use Task.FromResult, Task.Run or Task.Factory.StartNew:

FromResult:

public async Task DoWork()
{
       int res = await Task.FromResult<int>(GetSum(4, 5));	
}

private int GetSum(int a, int b)
{
	return a + b;
}

Please check out Stefan’s comments on the usage of FromResult in the comments section below the post.

Task.Run:

public async Task DoWork()
{
	Func<int> function = new Func<int>(() => GetSum(4, 5));
	int res = await Task.Run<int>(function);
}

private int GetSum(int a, int b)
{
	return a + b;
}

Task.Factory.StartNew:

public async Task DoWork()
{
	Func<int> function = new Func<int>(() => GetSum(4, 5));
	int res = await Task.Factory.StartNew<int>(function);			
}

private int GetSum(int a, int b)
{
	return a + b;
}

View the list of posts on the Task Parallel Library here.

.NET Developers’ user guide for troubleshooting networking problems Part 3

This is the last part in the series on basic networking for developers. Let’s look at firewalls first.

Firewalls

Firewalls are a common cause of port connectivity problems. What does a firewall do anyway? A firewall determines which connections are allowed to go through to the operating system and which ones are not. The firewall has a set of rules that state what traffic is allowed through. In the below example port 80 is let in but not port 25:

Firewall stop

Open Windows firewall as follows:

Open Windows firewall

This opens the Windows firewall manager:

Windows firewall managera

You’ll see a ling for Windows Firewall Properties somewhere in the middle of the screen:

Open windows firewall properties

Check the tabs in that window: you’ll see that you can set different options for the domain, private and public profiles which represent different states of Windows. It’s recommended to have the same settings for all 3 profiles unless you want to have different rules for your enterprise and home network. In this window you can turn the firewall on and off where the default is on.

Also by default we block all inbound connections and let all outbound traffic out. So traffic coming into our machine is blocked. You can also set the logging properties:

Open firewall logging settings

By default no dropped or successful connections are logged. If you suspect that your firewall drops data packets coming to your machine then it can be useful to log such events, so change that drop down list to yes. You can also specify the log file on the top of the window.

On the main Firewall screen you’ll see a link to the Inbound rules on the left hand side:

Open firewall inbound rules

You can add new inbound rules using the New Rule link:

Open firewall new inbound rule

You can create a rule by program, port, a predefined set of rules or a custom rule. For a program rule you specify an executable:

Specify Executable For Firewall Inbound Rule

This way we can open up or block the ports a specific program is listening on. Select some executable and click next. In the next screen you can select to open up the ports used by that executable or block them:

Inbound connection either blocked or allowed

Normally you’ll select the Allow option as all inbound traffic is blocked by default anyway. Click next and here you can define which profile to apply the rule to:

Which Windows profile to apply the rule to

As we said normally you’ll apply the same rules to all profiles. Then in the last step of the process you can provide a name for this rule. Give it some name and click finish. The new rule should appear in the list of rules on the main firewall screen.

This way of setting up a rule is useful if you’re not sure which port(s) a process uses. You can instead declare that all ports be opened up that are in use by that application.

Let’s create another rule, this time a port rule. Click the New Rule… link again and select the Port radio button and click next. On the next screen you’ll be able to define the type, i.e. TCP or UDP:

Inbound rule type tcp or udp

You can also define which port to open or close: all ports or just one specific or a range of ports. Let’s specify ’80, 443′ in the Specific local ports text box which will allow HTTP(S) traffic. Click Next and this screen will be familiar: you can allow or block the connection. Click Next. Again, the window will be familiar, you can define in which Windows profiles the rule will apply. In the last screen you can give a name to the rule, just like before. You’ll typically set this rule on your web server. If you don’t open up port 80 on your web server then no-one will be able to access the contents of your number one website.

You can add predefined rules by selecting the Predefined radio button in the very first window of the setup process. Open up the drop down list and you’ll see a whole bunch of predefined rules. These rules represent the Windows services that have been installed on your machine. You’ll see an option called Remote Desktop. This rule allows others to remotely connect to a computer. Click next and you’ll see some information on which port is going to be opened and some other parameters of the rule. If the predefined option needs more than one rule, such as Routing Volume Management, then all of them will be listed here.

The Custom rule type will give you a lot of freedom to define your rules. Click Next and you’ll see the window again where you can pick an application. Click Next to go to the Protocol and Ports window:

Protocol and ports inbound rule

Check out the Protocol type drop down list. Besides TCP and UDP which we discussed here you’ll see a whole range of other protocol types. E.g. the ICMPv4 protocol is used by the ping function by which you can ping a website in the command window. Select that protocol type. You can then click the Customize button where you can specify which ICMP packets to allow:

Customise ICMP packets

Select All ICMP types and click OK. Click Next to go to the Scope window. Here you can specify which local IP address this rule applies to and where we want to allow the traffic from – this is given in the remote IP section in the bottom half of the window. For now select the Any IP address option for both. The last three stages, i.e. Action, Profile and Name will all be familiar by now.

You can always come back later and update your rules. Just left-click a rule in the main window and select Properties from the context menu. This will open the Properties window:

Properties window for updating inbound rules

In this window you can specify a couple of options that were not available during the normal setup process. E.g. you can provide the authorised computers under the Computers tab. You can specify the users that are allowed to access this rule under the Users tab. Under the Scope tab you can define the IP addresses as we saw in the case of the Custom rule. As you see you can get to these options for any rule type but it’s offered during the setup phase only in the case of the Custom rule.

The Scope can be interesting if you want to setup the Remote Desktop predefined rule. You probably don’t want to let any computer get remote access to your computer, right. For any given machine in a network it is most likely enough to let other computers in the same subnet success it. E.g. I can remotely access the web servers that belong to the network of the company I work at. We don’t want anyone else to be able to access those computers.

In that scenario you can specify the correct IPs in the Remote IP address section:

Provide remote ips for remote access

The easiest way to achieve this is by the selecting the Predefined set of computers radio button and marking the local subnet option in the drop down list:

Select local subnet in remote access

You can allow other subnets as well by clicking the Add button again and filling in the IP ranges.

Network Address Translation and private IPs

We mentioned in the previous module of this series that with NAT we can have multiple private IPs corresponding to the same external public IP. We also said that we’re running short of public IP addresses so we can look at them as scarce and expensive commodities. Your ISP will probably only give you a single public IP although you can have several machines online in your home: your laptop, tablet, smart phone, your children’s computers etc. They will all have a private IP. Each private IP will be translated to the public IP in the outgoing traffic. Conversely the public IP will be translated to the correct private IP in the incoming traffic.

It actually makes sense that not all devices need public IPs. Why would anyone need to access your laptop from the public Internet?

The NAT device in the home environment is usually your router. It will translate the sessions back and forth between the internal and external addresses. In case you have a service on your home desktop that you want to make publicly available then the traffic coming to the your public IP will not be routed to the private IP of your desktop unless you set up a specific NAT rule on your routing device. This NAT rule will say that any inbound traffic coming to the external IP address be routed to a specific internal IP address.

Alternatively you can set up a port rule on the NAT device. The port rule will say that any traffic destined for a specific port be routed to an internal IP address. You can set up multiple port rules to direct incoming traffic to the correct private IP.

This can be useful if you want to host a website on your home desktop or you want to be able to remotely access a specific computer in your house.

The easiest way to find your public IP is use one of the many online IP services such as this or this. This sites will show you which IP address you’re coming from.

Thread safe collections in .NET: ConcurrentDictionary

Concurrent collections in .NET work very much like their single-thread counterparts with the difference that they are thread safe. These collections can be used in scenarios where you need to share a collection between Tasks. They are typed and use a lightweight synchronisation mechanism to ensure that they are safe and fast to use in parallel programming.

Concurrent dictionaries

Concurrent dictionaries are thread-safe equivalent collections of “normal” dictionaries, i.e. key-value pair collections. Concurrent dictionaries are ideal if you would like to share a key-pair collection of types K and V among several tasks.

Important methods:

  • TryAdd(K key, V value): adds an new key-value pair to the collection. Returns true if the insertion was successful
  • TryGetValue(K key, out V value): tries to retrieve the value of the key. Returns true if the extraction was successful and the value is assigned to the out parameter
  • TryRemove(K key, out V value): tries to remove the key-value pair associated with the key. Returns true if the deletion was successful and the value is assigned to the out parameter
  • TryUpdate(K key, V new, V current): update the value of the key-value pair with the ‘new’ value if the current value is equal to ‘current’. Returns true if the update was successful
  • ContainsKey(K key): same as ContainsKey of the normal Dictionary class, i.e. returns true if the key is found in the dictionary

The ‘try’ bit in the method names imply that your code needs to prepare for the event where the element could not be retrieved. If multiple threads retrieve elements from the same collection you cannot be sure what’s in there when a specific thread tries to read from it. E.g. even if ContainsKey returns true there’s no guarantee that the key-value pair is still present when the thread tries to read from it as another thread might have already removed it.

Example

We’ll need a simple object with a single property for the example:

class Series
{
	public int CurrentValue
	{
		get;
		set;
	}
}

The following code creates 20 tasks and each task increases the value of the CurrentValue property in the shared dictionary by 1000 in loop. So we’re expecting the final value to be 20000. We fill up the task array in a loop and start the tasks individually. The key-value in the dictionary may look like the following at a certain stage:

key: 0 (the task ID represented by the taskParameter object, which is the same as ‘i’ in the main loop), value: 40
key: 1, value 46
key: 2: value 43
.
.
.
key: 19, value 45

After the loop the values of each thread are added to the CurrentValue property:

Series series = new Series();
ConcurrentDictionary<int, int> concurrentDictionary = new ConcurrentDictionary<int, int>();
Task<int>[] taskArray = new Task<int>[20];
for (int i = 0; i < taskArray.Length; i++)
{
	concurrentDictionary.TryAdd(i, series.CurrentValue);

	taskArray[i] = Task.Factory.StartNew<int>((taskParameter) =>
	{
		int current;
		bool valueRetrieved;
		int key = Convert.ToInt32(taskParameter);
		for (int j = 0; j < 1000; j++)
		{
			valueRetrieved = concurrentDictionary.TryGetValue(key, out current);
			concurrentDictionary.TryUpdate(key, current + 1, current);
		}

		int result;
		valueRetrieved = concurrentDictionary.TryGetValue(key, out result);
		if (valueRetrieved)
		{
			return result;
		}
		else
		{
			throw new Exception(String.Format("No data item available for key {0}", taskParameter));
		}
	}, i);
}

for (int i = 0; i < taskArray.Length; i++)
{
	series.CurrentValue += taskArray[i].Result;
}

Console.WriteLine("Expected value {0}, Balance: {1}", 20000, series.CurrentValue);

View the list of posts on the Task Parallel Library here.

Thread safe collections in .NET: ConcurrentStack

Concurrent collections in .NET work very much like their single-thread counterparts with the difference that they are thread safe. These collections can be used in scenarios where you need to share a collection between Tasks. They are typed and use a lightweight synchronisation mechanism to ensure that they are safe and fast to use in parallel programming.

Concurrent stacks

If you don’t know what Stacks are then you can read about them here. The Stack of T generic collection has a thread-safe counterpart called ConcurrentStack. Important methods:

  • Push(T element): adds an item of type T to the collection
  • PushRange(T[] elements) and PushRange(T[] elements, int, int): same as Push but is used for adding an array of items to the collection
  • TryPeek(out T): tries to retrieve the next element from the collection without removing it. The value is set to the out parameter if the method succeeds. Otherwise it returns false.
  • TryPop(out T): tries to get the first element. It removes the item from the collection and sets the out parameter to the retrieved element. Otherwise the method returns false
  • TryPopRange(out T[] elements) and TryPopRange(out T[], int, int): same as TryPop but is used for arrays

The ‘try’ bit in the method names imply that your code needs to prepare for the event where the element could not be retrieved. If multiple threads retrieve elements from the same stack you cannot be sure what’s in there when a specific thread tries to read from it.

Example

Declare and fill a concurrent stack:

ConcurrentStack<int> concurrentStack = new ConcurrentStack<int>();

for (int i = 0; i < 5000; i++)
{
	concurrentStack.Push(i);
}

Next we’ll try to pop every item from the stack. The stack will be accessed by several tasks at the same time. The counter variable – which is also shared – will be used to check if all items have been retrieved.

int counter = 0;

Task[] stackTasks = new Task[10];
for (int i = 0; i < stackTasks.Length; i++)
{
	stackTasks[i] = Task.Factory.StartNew(() =>
	{
		while (concurrentStack.Count > 0)
		{
			int currentElement;
			bool success = concurrentStack.TryPop(out currentElement);
			if (success)
			{
				Interlocked.Increment(ref counter);
			}
		}
	});
}

The while loop will ensure that we’ll try to pop the items as long as there’s something left in the collection.

Wait for the tasks and print the number of items processed – the counter should have the same value as the number of items in the stack:

Task.WaitAll(stackTasks);
Console.WriteLine("Counter: {0}", counter);

View the list of posts on the Task Parallel Library here.

.NET Developers’ user guide for troubleshooting networking problems Part 2

We’ll continue our discussion on basic networking we started in the previous post.

IP routing

We’ll start off by looking at how traffic is routed from one network to another. Look at the following diagram:

Ip routing high level

We’ll talk about subnets in a little while: it’s a collection of computers that can talk to each other without needing to go through a router. A router connects different subnets – it routes traffic between different subnets. So if a computer within subnet A wants to talk to a computer on subnet B or on subnet C then traffic will pass through one or more routers. Let’s look at a couple of tools where we can watch this IP routing.

We’ll check out tracert – trace route – first which is a command line tool. Open up a command prompt and type tracert cnn.com:

Tracert cnn.com

The list is a lot longer, I didn’t copy the entire output. The list shows you the routing trace, i.e. the series of routers this traffic has to pass through in order to reach cnn.com. It takes measurements to see how long each hop takes.

Note that the values you see on your machine will almost certainly look different – the routing depends on your location in the world. It would be strange for you to have the same route as me if you are located in the US whereas I’m in Sweden.

The topmost entry is typically your local router connected to your modem. Then it goes on to the routers of my ISP which is Tele2 etc. You can then even read the geographic location of some of those routers: New York, Washington, Atlanta. The trace shows all the hops the traffic needs to pass from your computer to the cnn.com web server. The second column of millisecond values shows how long each hop took. In case of routing issues these values may be very large or you may even see a timeout.

If you identify a bad link in this chain then you’ll most likely have no control over it, you’ll just have to accept the news, but it can be good to be aware of the problems. The hops will take of course more time if you want to reach a server in the US from Europe. So if you have your business in the US and expect traffic from Europe then it can be a good idea to place a couple of web servers on the East Coast of the US so that these hops take shorter.

There’s another tool called pathping which has a similar purpose but gives you a more robust report. Type pathping cnn.com in the command window. The tool will output the same routing chain as tracert but will also perform a series of tests on these links over a long period of time. It will hit every link 100 times and output some statistics. You will see something like ‘Computing statistics for 400 seconds…’ in the command prompt meaning that it will take 400 seconds to calculate the statistics. The stats may look as follows:

Pathping output

What’s new is the lost/sent packet ratio: in the above case there were no packets lost whatsoever. This is what we should see in a healthy connection state.

Subnets

So routers direct traffic between subnets but what are subnets? The subnet is defined by a combination of the IP address and the subnet mask. Example:

IP: 193.169.115.230
Subnet mask: 255.255.255.000

The subnet mask has the same format as an IP address, i.e. it consists of 4 octets. The first 3 octets of the subnet mask have 8 bits turned on. 255 is written as 11111111 in binary notation, i.e. 8 bits. The last octet is turned off. The octets where the bits are turned on represent the network or the subnet. Where the bits are off, that represents a specific node on the subnet. In the above example the last octet of the IP address, i.e. 230 represents a specific node in the network denoted by 193.169.115. If the subnet mask is 255.255.000.000 then the the specific node is 115.230 within the subnet 193.169.

The subnet mask can be further broken down into 4*8 = 32 bits: 255 = 11111111 in the binary system as mentioned above, so the subnet example can also be written as 11111111.11111111.11111111.00000000. Therefore we have 24 bits turned on and 8 bits turned off. We can say that the subnet can have an IP range of 193.169.115.000 to 193.169.115.255. We can denote the same thing as 193.169.115.000/24 or 193.169.115.000/255.255.255.000. You can have a single 0 in place of the triple 0’s: 255.255.255.0.

Therefore if a computer with IP of 193.169.115.124 wants to communicate with another computer with IP 193.169.115.236 then the communication is direct, i.e. not routed through a router as both computers a located within the same subnet. If the other computer lies outside of that range then it will need to go through its default router. You can see how this changes if the subnet mask is 255.255.0.0 instead, i.e. 1111111.11111111.00000000.00000000. Then the IP range of this subnet becomes 193.169.0.0 to 193.169.255.255.

The subnet mask can vary and not always look that pretty: 255.255.254.0 i.e. 11111111.11111111.11111110.00000000. So we have 8 bits on, then 8 bits on then 7 bits on and 0 bits on. This is a 23 bit subnet which changes the IP range to 193.169.115.0 to 193.169.116.255. So you would normally think that the ranges are parts of different subnets, but you have to look at the subnet mask to be able to tell for sure.

Another example: with a subnet mask of 255.255.255.240 we have 28 bit subnet, i.e. 11111111.11111111.11111111.11110000. This is saying that we’ve broken down a ‘clean’ subnet into smaller pieces. The IP range will then span between 193.169.115.124 and 193.169.115.139. This is an extremely small subnet.

Route tables

How does the computer determine how to reach other subnets? This is where route tables enter the picture. Open a command prompt and enter the route PRINT command:

Route print command

Locate the first entry in the IPv4 route table with a network destination and subnet mask of 0.0.0.0 which means any IP address. The gateway the traffic needs to go through will be 192.168.0.254 on the interface 192.168.0.69 which is my current private IP address. The last value is the metric where the lowest value has priority so the gateway with the lowest metric will be the default one. The ‘on-link’ values are special: they denote your own computer so there’s no need for routing in those cases. E.g. all 127.x.x.x addresses point to your local machine, i.e. the localhost. Your computer will know the Gateway through the IP configuration:

IPConfig default gateway

Any traffic that’s destined to another network goes through this default gateway.

Network address translation (NAT)

You may have spotted the term ‘private IP’ in the previous section. There are 3 network ranges that are for private use only in IPv4: they cannot be routed to in the public internet. You typically get one public IP address from your Internet Service Provider but you can have several machines online at home: your PC, your desktop, your phone and possibly others. They each will use a private IP. IPconfig returned my private IP address under ‘IPv4 Address’.

A mechanism called Network address translation takes these private private IPs as they leave my home and converts them into the external public IP. It also translates the incoming public IP to the correct private IP address.

As private IPs are not reachable from the Internet it’s obvious that if you want to host a site available on the public Internet then you need a public IP address. You can actually host your website on your desktop at home by declaring that all traffic to your public IP address on port 80 – which standard HTTP traffic goes through – be routed to one specific private IP, in this case the private IP of your desktop. So you cannot direct port 80 traffic to more than one private IP.

The following ranges are for private use only:

  • 10.0.0.0 with a subnet mask of 255.0.0.0
  • 172.16.0.0 with a subnet mask of 255.240.0.0
  • 192.168.0.0 with a subnet mask of 255.255.0.0

You’ll recognise that the private IP I mentioned above fits in the the last range. The value you see in ipconfig on your machine will most certainly fit in one of these ranges.’

Ports

Ports are used to connect to a process on the server side by some protocol. The process will be listening to incoming messages on a certain port. HTTP websites listen to port 80 and HTTPS websites on port 443 by default. Many message-based products will listen on some default port: Apache Tomcat on port 8080, MongoDb on 27017, SQL server on 1433. The most common transport layer protocol is TCP which stands for Translation Control Protocol. Almost all web traffic – HTTP, mail – runs on TCP.

The sender, i.e. the client computer, wants to establish a session with the receiver, i.e. the server. The receiver will establish that session and declare that it’s ready to accept data. The client will then send one or more data sets. The server then sends a messaging confirming which messages it received. It’s possible that one or more data sets the client sends out is lost. In that case after a timeout period the lost data set will be resent. The server will confirm in case it received that message. The sender will know that the receiver has received the entire data pack:

Tcp diagram

The messaging process is managed by the networking stack. You don’t need to prepare anything extra in your application on the server side to accommodate the process.

TCP is not the only transport protocol type: UDP or User Datagram Protocol is another example. In UDP the sender doesn’t establish a session first. Instead, it starts sending data right away. Here there’s no built-in mechanism to resend lost data packets. So if some data set is lost then it cannot be resent:

UPD

UDP can be a good choice if losing some data packets is acceptable, e.g. in the case of video conferencing. If let’s say the 5th second of the video is lost and the participants keep talking then in the case of TCP the 5th second would be resent interrupting the flow of the video. Also, there’s no session involved in UDP meaning it has a lower overhead. However, in most messaging scenarios on the internet we do care about data and we need all data in order to process the requests. In that case TCP is the preferred choice.

You can test port connectivity using the command line using the telnet command. You can only test the TCP protocol this way. With UDP we simply send data and hope that it arrives. With the telnet command you can establish a session and send the commands to the receiving application, much like a web browser would do. Open a command prompt and type ‘telnet microsoft.com 80’: we want to connect to the process microsoft.com on port 80 which is the standard port for HTTP traffic. In case the command prompt is complaining about telnet not being an available command you need to turn on that feature:

Turn on telnet client

The command prompt should go all black upon a successful session setup:

Telnet microsoft.com

The microsoft.com server assumes that it has established a session with a browser and is ready to accept data. We could send HTTP GET requests to the server and expect some answer in return. Press Ctrl+C and enter to exit and you’ll see that the server has sent a 400 Bad Request:

Telnet HTTP bad request

The server didn’t understand what we wanted so it returned a HTTP 400. It even sent back some HTML that a web browser can render. We have successfully connected to an IIS process!

Now try to connect to port 81: type telnet microsoft.com 81 in the command prompt. There’s probably no process listening on port 81 on that web server but let’s see what happens:

Telnet connect timeout

The networking stack of the operating system is trying to establish a connection by sending out a session request to port 81 on microsoft.com. It’s possible that there’s some process listening on this port but the firewall is not letting through the request. Eventually we get the timeout message as seen above.

It’s not only HTTP websites that you can connect to of course using telnet but any type of process listening on some port. If you know that there’s an SQL server process on computer Machine01 then you could connect to that process and issue SQL commands by typing ‘telnet Machine01 1433’, where 1433 is the standard port SQL servers is expecting commands on.

Let’s now see how a mail server process responds. Let’s find the mail server name of gmail.com using nslookup:

NSlookup Gmail

Let’s try the one with the lowest preference value: gmail-smtp-in.l.google.com. SMTP mail traffic normally listens on port 25, so let’s issue the following telnet command: telnet gmail-smtp-in.l.google.com 25. If you successfully connect to the mail server then you should get a banner that says something like ‘220 mx.google.com ESMTP xxxx.79’. You can then send emailing commands to that port if you want to. You can quit the process by typing ‘quit’.

So you can use telnet if you know the port number to connect to. If you’re not sure then you can port scan the server using the the free nmap utility available here. Download the appropriate Windows installer and install the tool. Then you can issue the ‘nmap -v [machinename]’ command for a verbose port scan. The tool will try to connect to various TCP ports and list the ones where it was able to get through.

If you want to see which ports your computer is listening on then issue the ‘netstat -ano’ command:

netstat ano

The image shows only an extract of the full list of processes. 0.0.0.0 means that it’s going to listen on every IP address that’s available on the localhost. The port numbers are appended to the IP, e.g. :80, :443 etc. You’ll see the PID column on the right hand side. This shows the ID of the process or application that’s communicating with the process on that port. Open the task manager and add the PID column to the window:

Add process ID to task manager

You can then try and locate the process with some ID:

Task Manager with PID

This is helpful if you want to find a specific process using a port. Also, it helps finding conflicts when 2 or more processes are trying to listen on the same port.

Efficient linked lists in .NET

Sometimes you need a collection that’s modified a lot: you insert, update and remove items. A List of T is then inefficient as it needs to constantly rearrange all other items. The LinkedList class might be a better candidate.

A linked list is a doubly-linked list. Each item has a Next and Previous pointer to look at which element comes right before and after a particular object in the list. A linked list is very efficient at inserting and deleting items in particular.

Initialisation:

LinkedList<int> intList = new LinkedList<int>();

There are multiple ways to add a new item: AddAfter, AddBefore, AddFirst, AddLast.

Adding a new item on the top of the list:

intList.AddFirst(2);

Putting 3 ahead of 2:

intList.AddFirst(3);

It’s not possible to directly pull out an item with an indexer, such as [2].

You can, however, iterate through the collection:

foreach (var i in intList)
{
}

You can get a reference of the first item with the First property:

LinkedListNode<int> firstItem = intList.First;

You can then insert an item after that:

intList.AddAfter(first, 5);

This will add 5 in between 3 and 2.

Inserting before the first item is equally easy:

intList.AddBefore(first, 5);

You can get to the last item… can you guess how? Through the Last property.

The First and Last properties do not return an int, or the type that you provided in place of T. It returns a LinkedListNode of type T, which is int in this case. This object has a Previous and Next properties:

LinkedListNode<int> firstItem = intList.First;
firstItem.Previous;
firstItem.Next;

It also has a Value property which returns the actual value of the LinkedListNode object.

Another way of iterating through the list is the following:

LinkedListNode<int> item = intList.First;
while (item != null)
{
    int val = item.Value;
    item = item.Next;
}

Removing items can be done with methods such as RemoveLast(), RemoveFirst(), Remove(item).

.NET Developers’ user guide for troubleshooting networking problems Part 1

Introduction

As a programmer I normally don’t need to deal with hard-core networking issues in my job. The company I work at has a group of well-trained network engineers that fix network related problems for developers. However, I sometimes have the need to check some more basic things within networking to debug my code. Also, it can be beneficial to be able to follow along when network engineers discuss subnets, DNS records, ports and the like.

This is exactly the goal of this series: to help developers get to grips with the most basic concepts within networking. You certainly won’t become a professional networking engineer but you may not need that either.

Note: I did all demos on a Windows 7 machine. Other versions of Windows may output the values in a different format.

A network request

What happens when you enter a URL in your browser and press enter?

Networking diagram

The client wants to view http://www.bbc.co.uk to read the news so she enters that URL in the browser. The URL must then be converted into an IP address by the client computer therefore it needs to find out the IP address of http://www.bbc.co.uk. It performs this task by a service called DNS or Domain Naming System.

So it consults its configured DNS servers for the IP address of bbc.co.uk. The DNS server looks up the IP address and sends it back to the client. The client can now go out to the Internet through its switch and router and reach the data centre where the server is located. It will then pass through a firewall and switches to finally arrive at the web server. In the web server it enters the networking stack of the operating system, usually followed by a host based firewall and at last it reaches the process that’s the actual web server.

The data is then sent back to the client in the form of HTML, JSON, XML or whatever the format of the web application and it is rendered on the client machine.

The IP address

Each node in the network has an IP address, which is analogous to the unique address of your home. The postman needs to find you somehow so he will read the address on the letter and deliver it to your letterbox.

An IPv4 address is made up of 4 octets separated by a period similar to the following: 83.183.46.130.

Then we have the subnet mask which defines which part of the IP address is the subnet and which part is the specific node on that network. A subnet mask may look as follows: 255.255.255.0. We’ll look at subnets in a future post but for the time being it’s enough to know that if you try to reach an IP address which is not part of your subnet then it has to go through the default gateway. The default gateway can have an address such as 192.168.0.254.

Then we have the DNS servers that the client computer will use to turn names into IP addresses. Their IP typically looks like 75.75.75.75 or 75.75.75.76.

It’s easy to check your own IP configuration. Open a command prompt and run the ‘ipconfig’ command. The no-args version of the command will show your basic network configuration:

ip config no args

You will see the IPv4 address, the subnet mask and the default gateway. If you run the command ‘ipconfig -all’ then you’ll get a lot more information. You’ll see your host name at the top of the output. You’ll also find the DNS server somewhere in the middle. Your computer is configured to point to that DNS server to translate http://www.bbc.co.uk into numbers. Also, you’ll see something called the DHCP server. The DHCP server, which stands for Dynamic Host Configuration Protocol, is where your computer obtains the IP configuration.

So when a machine comes online and needs an IP configuration then it sends out a message asking for one. The DHCP server will catch that message and will respond with an IP address, a subnet mask, a default gateway and one or more DNS servers. The client machine will then take that information to configure itself and respond to the DHCP server saying that it will use that address. The DHCP server will then know that this IP is in use and will not hand it out to any other online machine for a specified period of time:

ObtainingAnAddressFromDHCP

Starting with Windows 2000 if the client is unable to get hold of an IP address then eventually it will give itself an address in the 169.254 address range which is a range owned by Microsoft. The client will eventually send out a message saying “I’m using 169.254.x.x”. This scenario occurs extremely rarely but if you see that your computer is struggling to get an IP and gets an IP in this range then it’s telling you that something is wrong and you’re not getting a response from the DHCP server.

What’s IPv6?

The current IP version is use is version 4, or IPv4. With the format mentioned above, i.e. 4 octets we get 2^32 – 2 raised to the power of 32 – different addresses. That’s quite a large number but is definitely finite and we’re soon reaching its upper limit.

IPv6 has been developed to extend the number of possible variations to 2^128 which is so large that we’ll enough left for all visiting extraterrestrials in the year of 10000.

Now IPv4 and IPv6 are running parallel. That’s why the ipconfig command gave you both and IPv4 and an IPv6 address. The ultimate goal is to only go forward with IPv6 sometime in the future.

You’ll see that the format of IPv6 is very different from IPv4. Example: 2001:0:5ef5:79fd:20df:3736:3f57:ffbe. As a developer you need to be aware of the differences if you need to log or validate an IP address or your app needs to show the new format on the screen.

DNS

So how is the name resolved that you enter in the URL text box of your web browser? As we mentioned above the client is configured to point to an initial DNS server. Say that it’s configured to contact nameserverA.isp.com. Therefore the client is going to ask this DNS server to resolve a URL and get the IP address belonging to that URL. The first DNS server probably won’t have this record so it sends a request to the root name servers: do you know where I can find this URL? The root name servers only contain the name server locations for the top level domains: .info, .com, .uk etc. and it’s the only thing it knows. So the root NS responds the first name server, like “no, I only know about top level domains but you can ask the .com name server because I know it has more information.” So nameserverA.isp.com asks the .com name server. The .com name server will have information about where to find the IP addresses of all .com URLs so it tells nameserverA.isp.com to go and ask the cnn.com name server. The cnn.com name server will have all the information about the cnn.com namespace and responds with the IP address.

DNS name resolution

The image is a bit messy so make sure you follow all the arrows based on the description. As you see the configured name server nameserverA.isp.com has a central role in the quest for finding the IP address. It takes a couple of stops before the final answer has been found.

NsLookup

You can use the command line tool called NsLookup to perform DNS queries. Let’s try to look up the IP address of cnn.com:

NsLookup cnn.com

Alternatively you can just type ‘nslookup’, press enter and then you can perform multiple queries:

NsLookup multiple queries

You’ll see that cnn.com returned more than one IP address. It means that we can reach cnn.com using several different IP addresses. Try http://www.microsoft.com and you’ll see that it’s been aliased to akadns.net, which are Akamai addresses. Akamai is a Content Delivery Network solution for faster downloads: Akamai homepage.

When you’re done using nslookup in the multiple query mode you can just type exit to come back to the ‘normal’ command prompt mode. In case you want to change the DNS server for your lookup query then enter the multiple query mode again by typing ‘nslookup’ and run the command ‘server [ip of the dns server you want to ask]’, e.g. server 123.456.678.43 and then ask for http://www.microsoft.com to see if you get the same IP address as in the case of the default DNS server.

The DNS records are cached for a certain period of time in the name servers to speed up the queries. Even your local machine caches this information. In your command window type ping the following 3 URLs using the ping command:

ping http://www.cnn.com
ping http://www.bbc.co.uk
ping http://www.microsoft.com

Then enter the following command: ipconfig /displaydns. This will bring up a list of all records cached on your local machine including the ones you have just pinged:

Ip config display DNS

Check out the Time to Live value. The record microsoft.com is cached for about 3500 seconds on my local machine. Wait a little bit and enter the ipconfig /displaydns command again. You should see that the Time to Live value should decrease. It will eventually reach 0 when the record is cleared from the cache.

Be aware of this caching feature as if you change a DNS record it will take some time to propagate it around the internet. Initially the old record will be returned from the DNS server as it is still in the cache.

Caching also means that if you ask for microsoft.com in your browser multiple times then there’s no need to go through the same name server lookup process over and over again. The immediate name server configured for your computer will have it in its cache and will be able to respond immediately with the correct IP address.

Override DNS in the local host file

It’s possible to override the DNS values on your local machine. This is done in the host file. On Windows machines it is usually located in the C:\Windows\System32\drivers\etc folder. The file is called hosts and you can open and edit it like a normal text file. You can add your ip-name pairs to the file using the following format:

Host file

So the format is: the ip address followed by a tab and then the name. You can even enter localhost IPs where localhost is always 127.0.0.1. You can add multiple names for the same IP as follows:

Host file with multiple names

You can enter the same made up values that I have and save the file. Go back to the command window and type ipconfig /displaydns again. Saving the hosts file will automatically clear the local cache which will be populated with the values in the host file. You should see the values you have just entered in the host file in the command window output. Run a ping command against one of the custom values in the host file, such as ping mysite.com and you’ll see that it will try to reach the IP that you specified. As that IP probably doesn’t exist it will just show a couple of Request timed out values.

Why would you modify the host file? If you migrate a website from one IP address to another, then you probably want to test the new environment in your browser, right? As the URL of the website doesn’t change then it will lead to the old IP address that exists in the name servers. You can then deploy the website to the new environment, override the host file and enter the URL again. You will then be directed to the IP you have specified in the host file. This is a very convenient solution for testing purposes: your clients will not see your beta site as they will still be directed to the old IP. Then when you’re done testing you can propagate the new IP value across the Internet.

Record types

When you type nslookup microsoft.com in the command prompt then it will provide you with one or more records of type A: an A record. An A record turns a name into an IP address. It is the default type of record that nslookup returns. There are other types of records and you can let nslookup return them as well. Run the nslookup command without specifying the name to enter the multiple query mode.

To query name server records you can set the type as follows:

NsLookup name server record

Here you see the name servers that are responsible for the microsoft.com namespace. Here we see 5 name servers. In the name resolution process your computer will pick one of those at random.

You can query mail exchange records (MX) but setting the type as follows:

set type=MX

Then query microsoft.com will give you something like this:

NsLookup mail exchange

If you send an email to Microsoft then you’ll send it to the microsoft-com.mail.protection.outlook.com mail server. That’s who will accept mail for the microsoft.com namespace. It’s possible that there are multiple mail servers in which case the preference parameter tells me in which order I should try to send the email.

Another record type is CNAME which stands for ‘canonical name’, it’s sort of an alias:

set type=CNAME

Then test microsoft.com. You’ll see no CNAME for that:

NSLookup no CNAME

The reason is that we cannot have a CNAME for the root of the domain. However, try http://www.microsoft.com, you’ll get a CNAME:

NsLookup with cname

http://www.microsoft.com is aliased to an Akamai address. This means that when you type http://www.microsoft.com in your web browser and get the IP address for it from the name server lookup then you will be directed to a server owned by the Akamai network.

The last record type to look at is the quad A, or AAAA record type. This is an IPv6 version of an A record so this turns the name into an IPv6 address. If you set the type to CNAME in the command prompt and query a name then you’ll get the AAAA records as well:

AAAA records

This is the case on with Windows 7. If you don’t see this output then test setting the type to AAAA first:

set type=AAAA

and then query a name.

These are the most common record types out there.

Using HashSet in .NET to allow unique values only

Hashsets allow you to add unique values into a collection. So if you have a rule that no two identical objects are added to a collection then a Set is a good choice.

Initialisation:

HashSet<int> integerSet = new HashSet<int>();

Add new items:

integerSet.Add(2);
integerSet.Add(3);

There’s no direct access available with an index parameter, like [2]. There’s no particular order to the items inserted to a Set. You’ll never know where the items end up and which one is the first in the list.

You can simply iterate through the values like this:

foreach (int value in integerSet)
{
}

If you try to add an integer that already exists then it will ignored. The Add method has a boolean return value. It will return false if the item you’re trying to add already exists in the collection and true otherwise.

You can easily build the intersection of two sets:

HashSet<int> integerSet1 = new HashSet<int>(){1, 2, 3};
HashSet<int> integerSet2 = new HashSet<int>(){2, 3, 4};

integerSet1.IntersectWith(integerSet2);

This operation will only keep those items in integerSet1 that were available in both sets, i.e. the intersection of the two sets: 2 and 3.

Building a union is equally easy:

integerSet1.UnionWith(integerSet2);

…resulting in integerSet1 = 1,2,3,4.

SymmetricExceptWith returns those items that are found in set 1 and set 2 but not both:

integerSet1.SymmetricExceptWith(integerSet2);

Result: 1,4.

If you want to store objects, such as Customer, Product, etc. then you need to take some extra care. Sets have no idea when two object are considered equal. By default the following operation will add both objects:

HashSet<Product> productSet = new HashSet<Product>();
productSet.Add(new Product() { Name = "A" });
productSet.Add(new Product() { Name = "A" });

These are two different objects, they point to two different locations in the memory heap.

One way to solve this is that object will implement the IEquatable interface and override the GetHashCode method:

public class Product : IEquatable<Product>
{
	public string Name { get; set; }

	public override int GetHashCode()
	{
		return Name.GetHashCode();
	}

	public bool Equals(Product other)
	{
		return this.Name.Equals(other.Name);
	}
}

This will prevent adding two equal objects to be added to the Set – provided that they should be considered equal of course.

LIFO data collection in .NET: Stack of T

If you need a data collection in .NET where you are forced to handle the objects in a last-in-first-out manner then you can use the Stack of T object: the last object added will be the first to come out. A typical scenario is a deck of cards: you pick the first card from the top, i.e. the one that was added last.

To initialise:

Stack&lt;Card&gt; stackOfCards = new Stack&lt;Card&gt;();

To add items:

stackOfCards.Push(new Card(ace));
stackOfCards.Push(new Card(ten));

To get the most recently added object from the stack:

Card next = stackOfCards.Pop();

The Pop() method will also remove the item from the collection. So the next time you call Pop() you’ll get the item added before the most recently added one.

Just like with a Queue of T you cannot directly reference an object in the stack collection by some index, e.g. [3].

The Peek() method will let you view the next item on the stack but it won’t be removed from the collection:

Card next = stackOfCards.Peek();

You can test for the absence of an item with Contains:

bool contains = stackOfCards.Contains(ace);

If you absolutely need to access the items directly then convert the stack into an array:

Card[] cardArray = stackOfCards.AsArray();

This will copy the items in the stack but leave the stack itself intact.

ultimatemindsettoday

A great WordPress.com site

Elliot Balynn's Blog

A directory of wonderful thoughts

HarsH ReaLiTy

A Good Blog is Hard to Find

Softwarearchitektur in der Praxis

Wissenswertes zu Webentwicklung, Domain-Driven Design und Microservices

Technology Talks

on Microsoft technologies, Web, Android and others

Software Engineering

Web development

Disparate Opinions

Various tidbits

chsakell's Blog

WEB APPLICATION DEVELOPMENT TUTORIALS WITH OPEN-SOURCE PROJECTS

Once Upon a Camayoc

Bite-size insight on Cyber Security for the not too technical.

Guru N Guns's

OneSolution To dOTnET.

Johnny Zraiby

Measuring programming progress by lines of code is like measuring aircraft building progress by weight.

%d bloggers like this: