.net | Exercises in .NET with Andras Nemes

Filter an array of integers with LINQ C# using a lambda expression

January 22, 2014 4 Comments

Consider the following array of integers:

int[] nums = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };

Say you want to filter out the odd numbers. One way to achieve this is using LINQ with a lambda expression. You’ll need a delegate that returns a boolean:

public delegate bool IntegerFilter(int i);

The following function accepts an integer array and a delegate and returns the filtered array:

public int[] FilterIntegerArray(int[] ints, IntegerFilter filter)
    {
      List<int> intList = new List<int>;
      foreach (int i in ints)
      {
        if (filter(i))
        {
          intList.Add(i);
        }
      }
      return intList.ToArray();
    }

You can call this method as follows using lambda expression that matches the signature of the delegate:

int[] oddNumbers =
        FilterIntegerArray(nums, i => ((i & 1) == 1) });

You can view all LINQ-related posts on this blog here.

Filed under .NET, LINQ Tagged with .net, c#, lambda, linq

Simple Func delegate example with LINQ in .NET C#

January 21, 2014 1 Comment

Say you’d like to filter an array of integers:

int[] ints = new int[] { 1, 2, 3, 4, 5, 6, 7, 8 };

You’re only interested in integers greater than 3. There are many ways to filter an array of T. One of them is using a Func delegate. Func delegates come in the following forms:

public delegate TR Func<TR>();
public delegate TR Func<Arg0, TR>(Arg0 a0);
public delegate TR Func<Arg0, Arg1, TR>(Arg0 a0, Arg1 a1);
.
.
.
public delegate TR Func<Arg0, Arg1, Arg2, Arg3, TR>(Arg0 a0, Arg1 a1, Arg2 a2, Arg3 a3);

…where TR is the return type of function and Arg(x) is the input parameter type. Note that the return type is the last element of the parameter types.

One of the overloaded versions of the LINQ Where operator accepts a Func delegate:

IEnumerable<T> Where<T>(this IEnumerable<T> source, Func<int, bool> predicate);

So we declare our Func:

Func<int, bool> GreaterThanTwo = i => i > 3;

Then we declare our query:

IEnumerable<int> intsGreaterThanTwo = ints.Where(GreaterThanTwo);

We finally enumerate the query:

foreach (int i in intsGreaterThanTwo)
{
        Console.WriteLine(i);
}

You can view all LINQ-related posts on this blog here.

Filed under .NET, LINQ Tagged with .net, c#, func, linq

Introduction to OAuth2: Json Web Tokens

January 20, 2014 13 Comments

Introduction

JSON web tokens are a sort of security token. As such, it is used for authentication purposes, and has similar attributes like the XLM-formatted SAML tokens we met in the series on Claims Bases Authentication. It shows the issuer of the token, the claims about the user, it must be signed to make it tamper-proof and it can have an expiration date. If you then log on to a web page then the authentication server will send back a security token that contains the data mentioned above. Upon successful authentication the web site will consume the token.

Again, like in the case of the SAML tokens there must be a trust relationship between the consumer and the issuer of the token. This ensures that even the contents of the token are trusted. The consumer knows about the key that the issuer uses to sign the token. That key can be used for validation purposes. At the time of writing this post SAML tokens are the most commonly used security tokens. SAML has quite a complex structure with a strict schema involved. It is very expressive with various encryption and signature options.

This last point is actually a limitation in the world of mobile devices. It’s needless to say how widespread mobile devices are in the world. There are probably more tablets and smartphones available nowadays than laptops and stationary computers. It feels like it’s only programmers like me who need computers… Mobile devices must perform their tasks with resources that are limited in comparison to computers. They are not well suited for parsing complex XML tokens.

This is where the JSON web tokens enter the scene with their simplified structure. The data is JSON which is more compact than XML and even mobile devices have the ability to parse them. JSON is also native to JavaScript which has grown into a very important language with applications such as Node.js, Windows 8 store apps, MongoDb etc. There are still a number of symmetric and asymmetric encryption options available for the token content and signature but it’s limited compared to what the SAML standard has to offer. However, the most widely accepted algorithms, such as RSA and AES are available and they will suit the vast majority of needs.

With the fast advance of mobile devices JSON web tokens will soon become – or already have? – the new standard for security tokens. They are already the de facto standard for OAuth2 and OpenID Connect in the modern mobile implementations.

Structure

A JSON security token consists of two parts:

Header with some metadata, e.g. on the algorithms and keys used to encrypt and sign the message
Claims

If you don’t know what claims are then make sure you understand the basics: start here.

These claims can be token specific, such as Issuer, Expiration date, or they can be defined by the application. You as the programmer are free to define what goes into the application-specific claims, such as FirstName, Email, NameOfPet, FavouriteColour, you name it.

If you’ve read through the series on Claims Based Auth available on this blog then the token specific claims are probably familiar to you:

Issuer: the identifier of the issuer of the token so that the recipient knows where the token is coming from
Audience: shows who the token is targeted at so that the recipient doesn’t use a token meant for a different application or website
IssuedAt: when the token was issued
Expiration: the expiration date of the token
Subject: typically a user ID that describes who the token is meant for

To keep the token size limited these claim types are abbreviated:

Issuer: iss
Audience: aud
IssuedAt: iat
Expiration: exp
Subject: sub

This is what a header can look like:

{
"typ": "JWT"
, "alg": "HS256"
}

Type: JWT means of course Json Web Token. Alg shows that the HMACSHA256 algorithm was used to sign the token.

The claims section can look like this:

{
"iss": "issuer name, usually some URL http://www.myauthenticationserver.com",
"exp": "a date in UNIX date format i.e. the number of seconds since 1970/01/01",
"aud": "the URL of the consumer likehttp: //www.mygreatsite.com",
"sub": "user identifier such as bobthebuilder"
}

Other claim types can be useful in the claims section: e.g. “client” to identify the application that requested the token. or “scope” to show the list of allowed operations for authorisation purposes:

{
"scope": ["read", "search"]
}

The list of claims can be extended as you wish, just as we saw in the case of SAML claims based authentication.

To make the token even more compact, the different sections are base64 encoded:

sdfgsdfgdfg.hrtg34twefwf4fg5g45gg.wsefefg345e4gf5g

These are just random characters, but locate the 2 periods in this string. They are delimiters for:

Header
Claims
Signature

…where each section is individually base64 encoded.

Some code

Open Visual Studio 2012 or higher, create a console application and add the below package from NuGet:

In addition, add a reference to the System.IdentityModel library.

There’s a number of ways to exchange JWT tokens between a sender and a receiver. In the below example I’ll use an RSACryptoServiceProvider to sign the JWT so that the receiver can validate it. If you don’t what RSA and asymmetric encryption mean then make sure to read upon it in the blog post mentioned above.

In short: the sender, i.e. the issuer of the token will have a pair of asymmetric encryption keys. The public key can be distributed to the receivers. The receiver will use the public key to validate the signature of the JWT token.

We won’t build a separate sender and receiver, that’s not the point here, but we want to simulate that the sender has access to both the private and public keys and the receiver only has the public key.

The following method will construct a valid RSA key pair:

private static RsaKeyGenerationResult GenerateRsaKeys()
{
	RSACryptoServiceProvider myRSA = new RSACryptoServiceProvider(2048);
	RSAParameters publicKey = myRSA.ExportParameters(true);
	RsaKeyGenerationResult result = new RsaKeyGenerationResult();
	result.PublicAndPrivateKey = myRSA.ToXmlString(true);
	result.PublicKeyOnly = myRSA.ToXmlString(false);
	return result;
}

…where RsaKeyGenerationResult is a DTO:

public class RsaKeyGenerationResult
{
	public string PublicKeyOnly { get; set; }
	public string PublicAndPrivateKey { get; set; }
}

In GenerateRsaKeys() we generate an RSA key and save its full set of keys and the public key only in two separate parameters of the RsaKeyGenerationResult object.

This is how to build and serialise the token:

RSACryptoServiceProvider publicAndPrivate = new RSACryptoServiceProvider();			
RsaKeyGenerationResult keyGenerationResult = GenerateRsaKeys();

publicAndPrivate.FromXmlString(keyGenerationResult.PublicAndPrivateKey);
JwtSecurityToken jwtToken = new JwtSecurityToken
	(issuer: "http://issuer.com", audience: "http://mysite.com"
	, claims: new List<Claim>() { new Claim(ClaimTypes.Name, "Andras Nemes") }
	, lifetime: new Lifetime(DateTime.UtcNow, DateTime.UtcNow.AddHours(1))
	, signingCredentials: new SigningCredentials(new RsaSecurityKey(publicAndPrivate)
		, SecurityAlgorithms.RsaSha256Signature, SecurityAlgorithms.Sha256Digest));

JwtSecurityTokenHandler tokenHandler = new JwtSecurityTokenHandler();
string tokenString = tokenHandler.WriteToken(jwtToken);

Console.WriteLine("Token string: {0}", tokenString);

You’ll recognise the parameters in the JwtSecurityToken constructor. You’ll see that we used the full RSA key to sign the token. The SecurityAlgorithms string enumeration stores the fully qualified names of the signature and digest algorithm. Those are compulsory arguments in the constructor.

Run the code up to this point and you should see a long string with the 3 segments mentioned above.

If your organisation has a valid X509 certificate then it can be used in place of the RsaSecurityKey object. Check out the X509AsymmetricSecurityKey object which accepts a 509Certificate2 object in its constructor. You’ll need to export the certificate with both the private and public keys and save it as a .pfx file. You can do that using the certmgr snap-in in Windows. Use the full file name of the .pfx file in the constructor of the X509Certificate2 constructor.

You can transmit the token in a number of different ways: in the header of an HTTP request, in the query string, in a cookie, etc. It’s now a string that you can send to the consumer.

The consumer can validate and read the claims from the token as follows:

JwtSecurityToken tokenReceived = new JwtSecurityToken(tokenString);

RSACryptoServiceProvider publicOnly = new RSACryptoServiceProvider();
publicOnly.FromXmlString(keyGenerationResult.PublicKeyOnly);
TokenValidationParameters validationParameters = new TokenValidationParameters()			
{
	ValidIssuer = "http://issuer.com"
	,AllowedAudience = "http://mysite.com"
	, SigningToken = new RsaSecurityToken(publicOnly)
};
			
JwtSecurityTokenHandler recipientTokenHandler = new JwtSecurityTokenHandler();
ClaimsPrincipal claimsPrincipal = recipientTokenHandler.ValidateToken(tokenReceived, validationParameters);

The client receives the base64 string which can be used in the constructor of the JwtSecurityToken object. We then use the public-key-only version of the RSACryptoServiceProvider to simulate that the receiver only has access to the public key of the sender. The TokenValidationParameters object can be used to build the validation logic. The JwtSecurityTokenHandler object is then used to validate the token. Upon successful validation you’ll get the ClaimsPrincipal which you’ll recognise from the posts on claims based auth mentioned above.

Set a breakpoint after the last line of code and inspect the contents of the claimsPrincipal object by hovering over it in VS.

The encoded JWT token can be decoded on this web page:

Read the next part in this series here.

You can view the list of posts on Security and Cryptography here.

Filed under .NET, Security Tagged with .net, c#, json web token, jwt, oauth2

Getting a return value from a Task with C#

January 17, 2014 15 Comments

Sometimes you want to get a return value from a Task as opposed to letting it run in the background and forgetting about it. You’ll need to specify the return type as a type parameter to the Task object: a Task of T.

.NET 4.0

Without specifying an input parameter:

Task<int> task = new Task<int>(() => 
{
    int total = 0;
    for (int i = 0; i < 500; i++)
    {
        total += i;
    }
    return total;
});

task.Start();
int result = Convert.ToInt32(task.Result);

We count to 500 and return the sum. The return value of the Task can be retrieved using the Result property which can be converted to the desired type.

You can provide an input parameter as well:

Task<int> task = new Task<int>(obj => 
{
    int total = 0;
    int max = (int)obj;
    for (int i = 0; i < max; i++)
    {
        total += i;
    }
    return total;
}, 300);

task.Start();
int result = Convert.ToInt32(task.Result);

We specify that we want to count to 300.

.NET 4.5

The recommended way in .NET 4.5 is to use Task.FromResult, Task.Run or Task.Factory.StartNew:

FromResult:

public async Task DoWork()
{
       int res = await Task.FromResult<int>(GetSum(4, 5));	
}

private int GetSum(int a, int b)
{
	return a + b;
}

Please check out Stefan’s comments on the usage of FromResult in the comments section below the post.

Task.Run:

public async Task DoWork()
{
	Func<int> function = new Func<int>(() => GetSum(4, 5));
	int res = await Task.Run<int>(function);
}

private int GetSum(int a, int b)
{
	return a + b;
}

Task.Factory.StartNew:

public async Task DoWork()
{
	Func<int> function = new Func<int>(() => GetSum(4, 5));
	int res = await Task.Factory.StartNew<int>(function);			
}

private int GetSum(int a, int b)
{
	return a + b;
}

View the list of posts on the Task Parallel Library here.

Filed under .NET, Task Parallel Library Tagged with .net, c#, parallelism, TPL

.NET Developers’ user guide for troubleshooting networking problems Part 3

January 16, 2014 Leave a comment

This is the last part in the series on basic networking for developers. Let’s look at firewalls first.

Firewalls

Firewalls are a common cause of port connectivity problems. What does a firewall do anyway? A firewall determines which connections are allowed to go through to the operating system and which ones are not. The firewall has a set of rules that state what traffic is allowed through. In the below example port 80 is let in but not port 25:

Open Windows firewall as follows:

This opens the Windows firewall manager:

You’ll see a ling for Windows Firewall Properties somewhere in the middle of the screen:

Check the tabs in that window: you’ll see that you can set different options for the domain, private and public profiles which represent different states of Windows. It’s recommended to have the same settings for all 3 profiles unless you want to have different rules for your enterprise and home network. In this window you can turn the firewall on and off where the default is on.

Also by default we block all inbound connections and let all outbound traffic out. So traffic coming into our machine is blocked. You can also set the logging properties:

By default no dropped or successful connections are logged. If you suspect that your firewall drops data packets coming to your machine then it can be useful to log such events, so change that drop down list to yes. You can also specify the log file on the top of the window.

On the main Firewall screen you’ll see a link to the Inbound rules on the left hand side:

You can add new inbound rules using the New Rule link:

You can create a rule by program, port, a predefined set of rules or a custom rule. For a program rule you specify an executable:

This way we can open up or block the ports a specific program is listening on. Select some executable and click next. In the next screen you can select to open up the ports used by that executable or block them:

Normally you’ll select the Allow option as all inbound traffic is blocked by default anyway. Click next and here you can define which profile to apply the rule to:

As we said normally you’ll apply the same rules to all profiles. Then in the last step of the process you can provide a name for this rule. Give it some name and click finish. The new rule should appear in the list of rules on the main firewall screen.

This way of setting up a rule is useful if you’re not sure which port(s) a process uses. You can instead declare that all ports be opened up that are in use by that application.

Let’s create another rule, this time a port rule. Click the New Rule… link again and select the Port radio button and click next. On the next screen you’ll be able to define the type, i.e. TCP or UDP:

You can also define which port to open or close: all ports or just one specific or a range of ports. Let’s specify ’80, 443′ in the Specific local ports text box which will allow HTTP(S) traffic. Click Next and this screen will be familiar: you can allow or block the connection. Click Next. Again, the window will be familiar, you can define in which Windows profiles the rule will apply. In the last screen you can give a name to the rule, just like before. You’ll typically set this rule on your web server. If you don’t open up port 80 on your web server then no-one will be able to access the contents of your number one website.

You can add predefined rules by selecting the Predefined radio button in the very first window of the setup process. Open up the drop down list and you’ll see a whole bunch of predefined rules. These rules represent the Windows services that have been installed on your machine. You’ll see an option called Remote Desktop. This rule allows others to remotely connect to a computer. Click next and you’ll see some information on which port is going to be opened and some other parameters of the rule. If the predefined option needs more than one rule, such as Routing Volume Management, then all of them will be listed here.

The Custom rule type will give you a lot of freedom to define your rules. Click Next and you’ll see the window again where you can pick an application. Click Next to go to the Protocol and Ports window:

Check out the Protocol type drop down list. Besides TCP and UDP which we discussed here you’ll see a whole range of other protocol types. E.g. the ICMPv4 protocol is used by the ping function by which you can ping a website in the command window. Select that protocol type. You can then click the Customize button where you can specify which ICMP packets to allow:

Select All ICMP types and click OK. Click Next to go to the Scope window. Here you can specify which local IP address this rule applies to and where we want to allow the traffic from – this is given in the remote IP section in the bottom half of the window. For now select the Any IP address option for both. The last three stages, i.e. Action, Profile and Name will all be familiar by now.

You can always come back later and update your rules. Just left-click a rule in the main window and select Properties from the context menu. This will open the Properties window:

In this window you can specify a couple of options that were not available during the normal setup process. E.g. you can provide the authorised computers under the Computers tab. You can specify the users that are allowed to access this rule under the Users tab. Under the Scope tab you can define the IP addresses as we saw in the case of the Custom rule. As you see you can get to these options for any rule type but it’s offered during the setup phase only in the case of the Custom rule.

The Scope can be interesting if you want to setup the Remote Desktop predefined rule. You probably don’t want to let any computer get remote access to your computer, right. For any given machine in a network it is most likely enough to let other computers in the same subnet success it. E.g. I can remotely access the web servers that belong to the network of the company I work at. We don’t want anyone else to be able to access those computers.

In that scenario you can specify the correct IPs in the Remote IP address section:

The easiest way to achieve this is by the selecting the Predefined set of computers radio button and marking the local subnet option in the drop down list:

You can allow other subnets as well by clicking the Add button again and filling in the IP ranges.

Network Address Translation and private IPs

We mentioned in the previous module of this series that with NAT we can have multiple private IPs corresponding to the same external public IP. We also said that we’re running short of public IP addresses so we can look at them as scarce and expensive commodities. Your ISP will probably only give you a single public IP although you can have several machines online in your home: your laptop, tablet, smart phone, your children’s computers etc. They will all have a private IP. Each private IP will be translated to the public IP in the outgoing traffic. Conversely the public IP will be translated to the correct private IP in the incoming traffic.

It actually makes sense that not all devices need public IPs. Why would anyone need to access your laptop from the public Internet?

The NAT device in the home environment is usually your router. It will translate the sessions back and forth between the internal and external addresses. In case you have a service on your home desktop that you want to make publicly available then the traffic coming to the your public IP will not be routed to the private IP of your desktop unless you set up a specific NAT rule on your routing device. This NAT rule will say that any inbound traffic coming to the external IP address be routed to a specific internal IP address.

Alternatively you can set up a port rule on the NAT device. The port rule will say that any traffic destined for a specific port be routed to an internal IP address. You can set up multiple port rules to direct incoming traffic to the correct private IP.

This can be useful if you want to host a website on your home desktop or you want to be able to remotely access a specific computer in your house.

The easiest way to find your public IP is use one of the many online IP services such as this or this. This sites will show you which IP address you’re coming from.

Filed under .NET, Networking Tagged with .net, networking

Thread safe collections in .NET: ConcurrentDictionary

January 15, 2014 4 Comments

Concurrent collections in .NET work very much like their single-thread counterparts with the difference that they are thread safe. These collections can be used in scenarios where you need to share a collection between Tasks. They are typed and use a lightweight synchronisation mechanism to ensure that they are safe and fast to use in parallel programming.

Concurrent dictionaries

Concurrent dictionaries are thread-safe equivalent collections of “normal” dictionaries, i.e. key-value pair collections. Concurrent dictionaries are ideal if you would like to share a key-pair collection of types K and V among several tasks.

Important methods:

TryAdd(K key, V value): adds an new key-value pair to the collection. Returns true if the insertion was successful
TryGetValue(K key, out V value): tries to retrieve the value of the key. Returns true if the extraction was successful and the value is assigned to the out parameter
TryRemove(K key, out V value): tries to remove the key-value pair associated with the key. Returns true if the deletion was successful and the value is assigned to the out parameter
TryUpdate(K key, V new, V current): update the value of the key-value pair with the ‘new’ value if the current value is equal to ‘current’. Returns true if the update was successful
ContainsKey(K key): same as ContainsKey of the normal Dictionary class, i.e. returns true if the key is found in the dictionary

The ‘try’ bit in the method names imply that your code needs to prepare for the event where the element could not be retrieved. If multiple threads retrieve elements from the same collection you cannot be sure what’s in there when a specific thread tries to read from it. E.g. even if ContainsKey returns true there’s no guarantee that the key-value pair is still present when the thread tries to read from it as another thread might have already removed it.

Example

We’ll need a simple object with a single property for the example:

class Series
{
	public int CurrentValue
	{
		get;
		set;
	}
}

The following code creates 20 tasks and each task increases the value of the CurrentValue property in the shared dictionary by 1000 in loop. So we’re expecting the final value to be 20000. We fill up the task array in a loop and start the tasks individually. The key-value in the dictionary may look like the following at a certain stage:

key: 0 (the task ID represented by the taskParameter object, which is the same as ‘i’ in the main loop), value: 40
key: 1, value 46
key: 2: value 43
.
.
.
key: 19, value 45

After the loop the values of each thread are added to the CurrentValue property:

Series series = new Series();
ConcurrentDictionary<int, int> concurrentDictionary = new ConcurrentDictionary<int, int>();
Task<int>[] taskArray = new Task<int>[20];
for (int i = 0; i < taskArray.Length; i++)
{
	concurrentDictionary.TryAdd(i, series.CurrentValue);

	taskArray[i] = Task.Factory.StartNew<int>((taskParameter) =>
	{
		int current;
		bool valueRetrieved;
		int key = Convert.ToInt32(taskParameter);
		for (int j = 0; j < 1000; j++)
		{
			valueRetrieved = concurrentDictionary.TryGetValue(key, out current);
			concurrentDictionary.TryUpdate(key, current + 1, current);
		}

		int result;
		valueRetrieved = concurrentDictionary.TryGetValue(key, out result);
		if (valueRetrieved)
		{
			return result;
		}
		else
		{
			throw new Exception(String.Format("No data item available for key {0}", taskParameter));
		}
	}, i);
}

for (int i = 0; i < taskArray.Length; i++)
{
	series.CurrentValue += taskArray[i].Result;
}

Console.WriteLine("Expected value {0}, Balance: {1}", 20000, series.CurrentValue);

View the list of posts on the Task Parallel Library here.

Filed under .NET, Task Parallel Library Tagged with .net, c#, parallelism, TPL

Thread safe collections in .NET: ConcurrentStack

January 14, 2014 Leave a comment

Concurrent stacks

If you don’t know what Stacks are then you can read about them here. The Stack of T generic collection has a thread-safe counterpart called ConcurrentStack. Important methods:

Push(T element): adds an item of type T to the collection
PushRange(T[] elements) and PushRange(T[] elements, int, int): same as Push but is used for adding an array of items to the collection
TryPeek(out T): tries to retrieve the next element from the collection without removing it. The value is set to the out parameter if the method succeeds. Otherwise it returns false.
TryPop(out T): tries to get the first element. It removes the item from the collection and sets the out parameter to the retrieved element. Otherwise the method returns false
TryPopRange(out T[] elements) and TryPopRange(out T[], int, int): same as TryPop but is used for arrays

The ‘try’ bit in the method names imply that your code needs to prepare for the event where the element could not be retrieved. If multiple threads retrieve elements from the same stack you cannot be sure what’s in there when a specific thread tries to read from it.

Example

Declare and fill a concurrent stack:

ConcurrentStack<int> concurrentStack = new ConcurrentStack<int>();

for (int i = 0; i < 5000; i++)
{
	concurrentStack.Push(i);
}

Next we’ll try to pop every item from the stack. The stack will be accessed by several tasks at the same time. The counter variable – which is also shared – will be used to check if all items have been retrieved.

int counter = 0;

Task[] stackTasks = new Task[10];
for (int i = 0; i < stackTasks.Length; i++)
{
	stackTasks[i] = Task.Factory.StartNew(() =>
	{
		while (concurrentStack.Count > 0)
		{
			int currentElement;
			bool success = concurrentStack.TryPop(out currentElement);
			if (success)
			{
				Interlocked.Increment(ref counter);
			}
		}
	});
}

The while loop will ensure that we’ll try to pop the items as long as there’s something left in the collection.

Wait for the tasks and print the number of items processed – the counter should have the same value as the number of items in the stack:

Task.WaitAll(stackTasks);
Console.WriteLine("Counter: {0}", counter);

View the list of posts on the Task Parallel Library here.

Filed under .NET, Task Parallel Library Tagged with .net, c#, parallelism, TPL

.NET Developers’ user guide for troubleshooting networking problems Part 2

January 13, 2014 1 Comment

We’ll continue our discussion on basic networking we started in the previous post.

IP routing

We’ll start off by looking at how traffic is routed from one network to another. Look at the following diagram:

We’ll talk about subnets in a little while: it’s a collection of computers that can talk to each other without needing to go through a router. A router connects different subnets – it routes traffic between different subnets. So if a computer within subnet A wants to talk to a computer on subnet B or on subnet C then traffic will pass through one or more routers. Let’s look at a couple of tools where we can watch this IP routing.

We’ll check out tracert – trace route – first which is a command line tool. Open up a command prompt and type tracert cnn.com:

The list is a lot longer, I didn’t copy the entire output. The list shows you the routing trace, i.e. the series of routers this traffic has to pass through in order to reach cnn.com. It takes measurements to see how long each hop takes.

Note that the values you see on your machine will almost certainly look different – the routing depends on your location in the world. It would be strange for you to have the same route as me if you are located in the US whereas I’m in Sweden.

The topmost entry is typically your local router connected to your modem. Then it goes on to the routers of my ISP which is Tele2 etc. You can then even read the geographic location of some of those routers: New York, Washington, Atlanta. The trace shows all the hops the traffic needs to pass from your computer to the cnn.com web server. The second column of millisecond values shows how long each hop took. In case of routing issues these values may be very large or you may even see a timeout.

If you identify a bad link in this chain then you’ll most likely have no control over it, you’ll just have to accept the news, but it can be good to be aware of the problems. The hops will take of course more time if you want to reach a server in the US from Europe. So if you have your business in the US and expect traffic from Europe then it can be a good idea to place a couple of web servers on the East Coast of the US so that these hops take shorter.

There’s another tool called pathping which has a similar purpose but gives you a more robust report. Type pathping cnn.com in the command window. The tool will output the same routing chain as tracert but will also perform a series of tests on these links over a long period of time. It will hit every link 100 times and output some statistics. You will see something like ‘Computing statistics for 400 seconds…’ in the command prompt meaning that it will take 400 seconds to calculate the statistics. The stats may look as follows:

What’s new is the lost/sent packet ratio: in the above case there were no packets lost whatsoever. This is what we should see in a healthy connection state.

Subnets

So routers direct traffic between subnets but what are subnets? The subnet is defined by a combination of the IP address and the subnet mask. Example:

IP: 193.169.115.230
Subnet mask: 255.255.255.000

The subnet mask has the same format as an IP address, i.e. it consists of 4 octets. The first 3 octets of the subnet mask have 8 bits turned on. 255 is written as 11111111 in binary notation, i.e. 8 bits. The last octet is turned off. The octets where the bits are turned on represent the network or the subnet. Where the bits are off, that represents a specific node on the subnet. In the above example the last octet of the IP address, i.e. 230 represents a specific node in the network denoted by 193.169.115. If the subnet mask is 255.255.000.000 then the the specific node is 115.230 within the subnet 193.169.

The subnet mask can be further broken down into 4*8 = 32 bits: 255 = 11111111 in the binary system as mentioned above, so the subnet example can also be written as 11111111.11111111.11111111.00000000. Therefore we have 24 bits turned on and 8 bits turned off. We can say that the subnet can have an IP range of 193.169.115.000 to 193.169.115.255. We can denote the same thing as 193.169.115.000/24 or 193.169.115.000/255.255.255.000. You can have a single 0 in place of the triple 0’s: 255.255.255.0.

Therefore if a computer with IP of 193.169.115.124 wants to communicate with another computer with IP 193.169.115.236 then the communication is direct, i.e. not routed through a router as both computers a located within the same subnet. If the other computer lies outside of that range then it will need to go through its default router. You can see how this changes if the subnet mask is 255.255.0.0 instead, i.e. 1111111.11111111.00000000.00000000. Then the IP range of this subnet becomes 193.169.0.0 to 193.169.255.255.

The subnet mask can vary and not always look that pretty: 255.255.254.0 i.e. 11111111.11111111.11111110.00000000. So we have 8 bits on, then 8 bits on then 7 bits on and 0 bits on. This is a 23 bit subnet which changes the IP range to 193.169.115.0 to 193.169.116.255. So you would normally think that the ranges are parts of different subnets, but you have to look at the subnet mask to be able to tell for sure.

Another example: with a subnet mask of 255.255.255.240 we have 28 bit subnet, i.e. 11111111.11111111.11111111.11110000. This is saying that we’ve broken down a ‘clean’ subnet into smaller pieces. The IP range will then span between 193.169.115.124 and 193.169.115.139. This is an extremely small subnet.

Route tables

How does the computer determine how to reach other subnets? This is where route tables enter the picture. Open a command prompt and enter the route PRINT command:

Locate the first entry in the IPv4 route table with a network destination and subnet mask of 0.0.0.0 which means any IP address. The gateway the traffic needs to go through will be 192.168.0.254 on the interface 192.168.0.69 which is my current private IP address. The last value is the metric where the lowest value has priority so the gateway with the lowest metric will be the default one. The ‘on-link’ values are special: they denote your own computer so there’s no need for routing in those cases. E.g. all 127.x.x.x addresses point to your local machine, i.e. the localhost. Your computer will know the Gateway through the IP configuration:

Any traffic that’s destined to another network goes through this default gateway.

Network address translation (NAT)

You may have spotted the term ‘private IP’ in the previous section. There are 3 network ranges that are for private use only in IPv4: they cannot be routed to in the public internet. You typically get one public IP address from your Internet Service Provider but you can have several machines online at home: your PC, your desktop, your phone and possibly others. They each will use a private IP. IPconfig returned my private IP address under ‘IPv4 Address’.

A mechanism called Network address translation takes these private private IPs as they leave my home and converts them into the external public IP. It also translates the incoming public IP to the correct private IP address.

As private IPs are not reachable from the Internet it’s obvious that if you want to host a site available on the public Internet then you need a public IP address. You can actually host your website on your desktop at home by declaring that all traffic to your public IP address on port 80 – which standard HTTP traffic goes through – be routed to one specific private IP, in this case the private IP of your desktop. So you cannot direct port 80 traffic to more than one private IP.

The following ranges are for private use only:

10.0.0.0 with a subnet mask of 255.0.0.0
172.16.0.0 with a subnet mask of 255.240.0.0
192.168.0.0 with a subnet mask of 255.255.0.0

You’ll recognise that the private IP I mentioned above fits in the the last range. The value you see in ipconfig on your machine will most certainly fit in one of these ranges.’

Ports

Ports are used to connect to a process on the server side by some protocol. The process will be listening to incoming messages on a certain port. HTTP websites listen to port 80 and HTTPS websites on port 443 by default. Many message-based products will listen on some default port: Apache Tomcat on port 8080, MongoDb on 27017, SQL server on 1433. The most common transport layer protocol is TCP which stands for Translation Control Protocol. Almost all web traffic – HTTP, mail – runs on TCP.

The sender, i.e. the client computer, wants to establish a session with the receiver, i.e. the server. The receiver will establish that session and declare that it’s ready to accept data. The client will then send one or more data sets. The server then sends a messaging confirming which messages it received. It’s possible that one or more data sets the client sends out is lost. In that case after a timeout period the lost data set will be resent. The server will confirm in case it received that message. The sender will know that the receiver has received the entire data pack:

The messaging process is managed by the networking stack. You don’t need to prepare anything extra in your application on the server side to accommodate the process.

TCP is not the only transport protocol type: UDP or User Datagram Protocol is another example. In UDP the sender doesn’t establish a session first. Instead, it starts sending data right away. Here there’s no built-in mechanism to resend lost data packets. So if some data set is lost then it cannot be resent:

UDP can be a good choice if losing some data packets is acceptable, e.g. in the case of video conferencing. If let’s say the 5th second of the video is lost and the participants keep talking then in the case of TCP the 5th second would be resent interrupting the flow of the video. Also, there’s no session involved in UDP meaning it has a lower overhead. However, in most messaging scenarios on the internet we do care about data and we need all data in order to process the requests. In that case TCP is the preferred choice.

You can test port connectivity using the command line using the telnet command. You can only test the TCP protocol this way. With UDP we simply send data and hope that it arrives. With the telnet command you can establish a session and send the commands to the receiving application, much like a web browser would do. Open a command prompt and type ‘telnet microsoft.com 80’: we want to connect to the process microsoft.com on port 80 which is the standard port for HTTP traffic. In case the command prompt is complaining about telnet not being an available command you need to turn on that feature:

The command prompt should go all black upon a successful session setup:

The microsoft.com server assumes that it has established a session with a browser and is ready to accept data. We could send HTTP GET requests to the server and expect some answer in return. Press Ctrl+C and enter to exit and you’ll see that the server has sent a 400 Bad Request:

The server didn’t understand what we wanted so it returned a HTTP 400. It even sent back some HTML that a web browser can render. We have successfully connected to an IIS process!

Now try to connect to port 81: type telnet microsoft.com 81 in the command prompt. There’s probably no process listening on port 81 on that web server but let’s see what happens:

The networking stack of the operating system is trying to establish a connection by sending out a session request to port 81 on microsoft.com. It’s possible that there’s some process listening on this port but the firewall is not letting through the request. Eventually we get the timeout message as seen above.

It’s not only HTTP websites that you can connect to of course using telnet but any type of process listening on some port. If you know that there’s an SQL server process on computer Machine01 then you could connect to that process and issue SQL commands by typing ‘telnet Machine01 1433’, where 1433 is the standard port SQL servers is expecting commands on.

Let’s now see how a mail server process responds. Let’s find the mail server name of gmail.com using nslookup:

Let’s try the one with the lowest preference value: gmail-smtp-in.l.google.com. SMTP mail traffic normally listens on port 25, so let’s issue the following telnet command: telnet gmail-smtp-in.l.google.com 25. If you successfully connect to the mail server then you should get a banner that says something like ‘220 mx.google.com ESMTP xxxx.79’. You can then send emailing commands to that port if you want to. You can quit the process by typing ‘quit’.

So you can use telnet if you know the port number to connect to. If you’re not sure then you can port scan the server using the the free nmap utility available here. Download the appropriate Windows installer and install the tool. Then you can issue the ‘nmap -v [machinename]’ command for a verbose port scan. The tool will try to connect to various TCP ports and list the ones where it was able to get through.

If you want to see which ports your computer is listening on then issue the ‘netstat -ano’ command:

The image shows only an extract of the full list of processes. 0.0.0.0 means that it’s going to listen on every IP address that’s available on the localhost. The port numbers are appended to the IP, e.g. :80, :443 etc. You’ll see the PID column on the right hand side. This shows the ID of the process or application that’s communicating with the process on that port. Open the task manager and add the PID column to the window:

You can then try and locate the process with some ID:

This is helpful if you want to find a specific process using a port. Also, it helps finding conflicts when 2 or more processes are trying to listen on the same port.

Filed under .NET, Networking Tagged with .net, networking

Efficient linked lists in .NET

January 10, 2014 Leave a comment

Sometimes you need a collection that’s modified a lot: you insert, update and remove items. A List of T is then inefficient as it needs to constantly rearrange all other items. The LinkedList class might be a better candidate.

A linked list is a doubly-linked list. Each item has a Next and Previous pointer to look at which element comes right before and after a particular object in the list. A linked list is very efficient at inserting and deleting items in particular.

Initialisation:

LinkedList<int> intList = new LinkedList<int>();

There are multiple ways to add a new item: AddAfter, AddBefore, AddFirst, AddLast.

Adding a new item on the top of the list:

intList.AddFirst(2);

Putting 3 ahead of 2:

intList.AddFirst(3);

It’s not possible to directly pull out an item with an indexer, such as [2].

You can, however, iterate through the collection:

foreach (var i in intList)
{
}

You can get a reference of the first item with the First property:

LinkedListNode<int> firstItem = intList.First;

You can then insert an item after that:

intList.AddAfter(first, 5);

This will add 5 in between 3 and 2.

Inserting before the first item is equally easy:

intList.AddBefore(first, 5);

You can get to the last item… can you guess how? Through the Last property.

The First and Last properties do not return an int, or the type that you provided in place of T. It returns a LinkedListNode of type T, which is int in this case. This object has a Previous and Next properties:

LinkedListNode<int> firstItem = intList.First;
firstItem.Previous;
firstItem.Next;

It also has a Value property which returns the actual value of the LinkedListNode object.

Another way of iterating through the list is the following:

LinkedListNode<int> item = intList.First;
while (item != null)
{
    int val = item.Value;
    item = item.Next;
}

Removing items can be done with methods such as RemoveLast(), RemoveFirst(), Remove(item).

Filed under .NET, Generics Tagged with .net, c#, collections, generics

.NET Developers’ user guide for troubleshooting networking problems Part 1

January 9, 2014 Leave a comment

Introduction

As a programmer I normally don’t need to deal with hard-core networking issues in my job. The company I work at has a group of well-trained network engineers that fix network related problems for developers. However, I sometimes have the need to check some more basic things within networking to debug my code. Also, it can be beneficial to be able to follow along when network engineers discuss subnets, DNS records, ports and the like.

This is exactly the goal of this series: to help developers get to grips with the most basic concepts within networking. You certainly won’t become a professional networking engineer but you may not need that either.

Note: I did all demos on a Windows 7 machine. Other versions of Windows may output the values in a different format.

A network request

What happens when you enter a URL in your browser and press enter?

The client wants to view http://www.bbc.co.uk to read the news so she enters that URL in the browser. The URL must then be converted into an IP address by the client computer therefore it needs to find out the IP address of http://www.bbc.co.uk. It performs this task by a service called DNS or Domain Naming System.

So it consults its configured DNS servers for the IP address of bbc.co.uk. The DNS server looks up the IP address and sends it back to the client. The client can now go out to the Internet through its switch and router and reach the data centre where the server is located. It will then pass through a firewall and switches to finally arrive at the web server. In the web server it enters the networking stack of the operating system, usually followed by a host based firewall and at last it reaches the process that’s the actual web server.

The data is then sent back to the client in the form of HTML, JSON, XML or whatever the format of the web application and it is rendered on the client machine.

The IP address

Each node in the network has an IP address, which is analogous to the unique address of your home. The postman needs to find you somehow so he will read the address on the letter and deliver it to your letterbox.

An IPv4 address is made up of 4 octets separated by a period similar to the following: 83.183.46.130.

Then we have the subnet mask which defines which part of the IP address is the subnet and which part is the specific node on that network. A subnet mask may look as follows: 255.255.255.0. We’ll look at subnets in a future post but for the time being it’s enough to know that if you try to reach an IP address which is not part of your subnet then it has to go through the default gateway. The default gateway can have an address such as 192.168.0.254.

Then we have the DNS servers that the client computer will use to turn names into IP addresses. Their IP typically looks like 75.75.75.75 or 75.75.75.76.

It’s easy to check your own IP configuration. Open a command prompt and run the ‘ipconfig’ command. The no-args version of the command will show your basic network configuration:

You will see the IPv4 address, the subnet mask and the default gateway. If you run the command ‘ipconfig -all’ then you’ll get a lot more information. You’ll see your host name at the top of the output. You’ll also find the DNS server somewhere in the middle. Your computer is configured to point to that DNS server to translate http://www.bbc.co.uk into numbers. Also, you’ll see something called the DHCP server. The DHCP server, which stands for Dynamic Host Configuration Protocol, is where your computer obtains the IP configuration.

So when a machine comes online and needs an IP configuration then it sends out a message asking for one. The DHCP server will catch that message and will respond with an IP address, a subnet mask, a default gateway and one or more DNS servers. The client machine will then take that information to configure itself and respond to the DHCP server saying that it will use that address. The DHCP server will then know that this IP is in use and will not hand it out to any other online machine for a specified period of time:

Starting with Windows 2000 if the client is unable to get hold of an IP address then eventually it will give itself an address in the 169.254 address range which is a range owned by Microsoft. The client will eventually send out a message saying “I’m using 169.254.x.x”. This scenario occurs extremely rarely but if you see that your computer is struggling to get an IP and gets an IP in this range then it’s telling you that something is wrong and you’re not getting a response from the DHCP server.

What’s IPv6?

The current IP version is use is version 4, or IPv4. With the format mentioned above, i.e. 4 octets we get 2^32 – 2 raised to the power of 32 – different addresses. That’s quite a large number but is definitely finite and we’re soon reaching its upper limit.

IPv6 has been developed to extend the number of possible variations to 2^128 which is so large that we’ll enough left for all visiting extraterrestrials in the year of 10000.

Now IPv4 and IPv6 are running parallel. That’s why the ipconfig command gave you both and IPv4 and an IPv6 address. The ultimate goal is to only go forward with IPv6 sometime in the future.

You’ll see that the format of IPv6 is very different from IPv4. Example: 2001:0:5ef5:79fd:20df:3736:3f57:ffbe. As a developer you need to be aware of the differences if you need to log or validate an IP address or your app needs to show the new format on the screen.

DNS

So how is the name resolved that you enter in the URL text box of your web browser? As we mentioned above the client is configured to point to an initial DNS server. Say that it’s configured to contact nameserverA.isp.com. Therefore the client is going to ask this DNS server to resolve a URL and get the IP address belonging to that URL. The first DNS server probably won’t have this record so it sends a request to the root name servers: do you know where I can find this URL? The root name servers only contain the name server locations for the top level domains: .info, .com, .uk etc. and it’s the only thing it knows. So the root NS responds the first name server, like “no, I only know about top level domains but you can ask the .com name server because I know it has more information.” So nameserverA.isp.com asks the .com name server. The .com name server will have information about where to find the IP addresses of all .com URLs so it tells nameserverA.isp.com to go and ask the cnn.com name server. The cnn.com name server will have all the information about the cnn.com namespace and responds with the IP address.

The image is a bit messy so make sure you follow all the arrows based on the description. As you see the configured name server nameserverA.isp.com has a central role in the quest for finding the IP address. It takes a couple of stops before the final answer has been found.

NsLookup

You can use the command line tool called NsLookup to perform DNS queries. Let’s try to look up the IP address of cnn.com:

Alternatively you can just type ‘nslookup’, press enter and then you can perform multiple queries:

You’ll see that cnn.com returned more than one IP address. It means that we can reach cnn.com using several different IP addresses. Try http://www.microsoft.com and you’ll see that it’s been aliased to akadns.net, which are Akamai addresses. Akamai is a Content Delivery Network solution for faster downloads: Akamai homepage.

When you’re done using nslookup in the multiple query mode you can just type exit to come back to the ‘normal’ command prompt mode. In case you want to change the DNS server for your lookup query then enter the multiple query mode again by typing ‘nslookup’ and run the command ‘server [ip of the dns server you want to ask]’, e.g. server 123.456.678.43 and then ask for http://www.microsoft.com to see if you get the same IP address as in the case of the default DNS server.

The DNS records are cached for a certain period of time in the name servers to speed up the queries. Even your local machine caches this information. In your command window type ping the following 3 URLs using the ping command:

ping http://www.cnn.com
ping http://www.bbc.co.uk
ping http://www.microsoft.com

Then enter the following command: ipconfig /displaydns. This will bring up a list of all records cached on your local machine including the ones you have just pinged:

Check out the Time to Live value. The record microsoft.com is cached for about 3500 seconds on my local machine. Wait a little bit and enter the ipconfig /displaydns command again. You should see that the Time to Live value should decrease. It will eventually reach 0 when the record is cleared from the cache.

Be aware of this caching feature as if you change a DNS record it will take some time to propagate it around the internet. Initially the old record will be returned from the DNS server as it is still in the cache.

Caching also means that if you ask for microsoft.com in your browser multiple times then there’s no need to go through the same name server lookup process over and over again. The immediate name server configured for your computer will have it in its cache and will be able to respond immediately with the correct IP address.

Override DNS in the local host file

It’s possible to override the DNS values on your local machine. This is done in the host file. On Windows machines it is usually located in the C:\Windows\System32\drivers\etc folder. The file is called hosts and you can open and edit it like a normal text file. You can add your ip-name pairs to the file using the following format:

So the format is: the ip address followed by a tab and then the name. You can even enter localhost IPs where localhost is always 127.0.0.1. You can add multiple names for the same IP as follows:

You can enter the same made up values that I have and save the file. Go back to the command window and type ipconfig /displaydns again. Saving the hosts file will automatically clear the local cache which will be populated with the values in the host file. You should see the values you have just entered in the host file in the command window output. Run a ping command against one of the custom values in the host file, such as ping mysite.com and you’ll see that it will try to reach the IP that you specified. As that IP probably doesn’t exist it will just show a couple of Request timed out values.

Why would you modify the host file? If you migrate a website from one IP address to another, then you probably want to test the new environment in your browser, right? As the URL of the website doesn’t change then it will lead to the old IP address that exists in the name servers. You can then deploy the website to the new environment, override the host file and enter the URL again. You will then be directed to the IP you have specified in the host file. This is a very convenient solution for testing purposes: your clients will not see your beta site as they will still be directed to the old IP. Then when you’re done testing you can propagate the new IP value across the Internet.

Record types

When you type nslookup microsoft.com in the command prompt then it will provide you with one or more records of type A: an A record. An A record turns a name into an IP address. It is the default type of record that nslookup returns. There are other types of records and you can let nslookup return them as well. Run the nslookup command without specifying the name to enter the multiple query mode.

To query name server records you can set the type as follows:

Here you see the name servers that are responsible for the microsoft.com namespace. Here we see 5 name servers. In the name resolution process your computer will pick one of those at random.

You can query mail exchange records (MX) but setting the type as follows:

set type=MX

Then query microsoft.com will give you something like this:

If you send an email to Microsoft then you’ll send it to the microsoft-com.mail.protection.outlook.com mail server. That’s who will accept mail for the microsoft.com namespace. It’s possible that there are multiple mail servers in which case the preference parameter tells me in which order I should try to send the email.

Another record type is CNAME which stands for ‘canonical name’, it’s sort of an alias:

set type=CNAME

Then test microsoft.com. You’ll see no CNAME for that:

The reason is that we cannot have a CNAME for the root of the domain. However, try http://www.microsoft.com, you’ll get a CNAME:

http://www.microsoft.com is aliased to an Akamai address. This means that when you type http://www.microsoft.com in your web browser and get the IP address for it from the name server lookup then you will be directed to a server owned by the Akamai network.

The last record type to look at is the quad A, or AAAA record type. This is an IPv6 version of an A record so this turns the name into an IPv6 address. If you set the type to CNAME in the command prompt and query a name then you’ll get the AAAA records as well:

This is the case on with Windows 7. If you don’t see this output then test setting the type to AAAA first:

set type=AAAA

and then query a name.

These are the most common record types out there.

Filed under .NET, Networking Tagged with .net, networking

← Older posts

Newer posts →

Exercises in .NET with Andras Nemes

Filter an array of integers with LINQ C# using a lambda expression

Simple Func delegate example with LINQ in .NET C#

Introduction to OAuth2: Json Web Tokens

Getting a return value from a Task with C#

.NET Developers’ user guide for troubleshooting networking problems Part 3

Thread safe collections in .NET: ConcurrentDictionary

Thread safe collections in .NET: ConcurrentStack

.NET Developers’ user guide for troubleshooting networking problems Part 2

Efficient linked lists in .NET

.NET Developers’ user guide for troubleshooting networking problems Part 1

My profile

Andras Nemes

Personal Links

Verified Services

Follow my blog via email

Top Posts & Pages

History

My tweets

Blogs I Follow

Share:

Like this:

Share:

Like this:

Share:

Like this:

Share:

Like this:

Share:

Like this:

Share:

Like this:

Share:

Like this:

Share:

Like this:

Share:

Like this:

Share:

Like this:

My profile

Personal Links

Verified Services

Follow my blog via email

Top Posts & Pages

History

Keywords

Blogs I Follow