Introduction to forms based authentication in ASP.NET MVC5 Part 3

Introduction

In the previous post we started digging into the components around Identity in MVC5. We looked at the default database and the Microsoft.AspNet.Identity.Core. We also saw how the MVC5 template with Forms based authentication creates a default UserManager class which can handle user creation, login, updates, claims etc.

We’ll continue our discussion with some other components around Identity in an MVC5 project.

We’ll build on the demo application from the previous two parts of this series.

EntityFramework

EntityFramework is the default choice for storing users in an MVC5 project. As far as identity is concerned EF is encapsulated in the Microsoft.AspNet.Identity.EntityFramework package. Do you recall the interfaces around user management in the Core assembly? IUserStore, IUserLoginStore, IUserClaimsStore etc. The EntityFramework assembly provides the out-of-the-box concrete implementation of each of those abstractions. Examples:

  • IdentityUser implements IUser
  • IdentityRole implements IRole
  • UserStore implements IUserStore, IUserLoginStore, IUserClaimsStore etc., so it contains a large amount of methods and properties which are implementations of the interfaces

IdentityUser and IdentityRole are entities that depend on EF for their persistence mechanism. Recall the database that MVC5 created for us in the local DB. Those are the DB representations of these identities. In AccountController.cs this UserStore is passed to the constructor of UserManager which in turn is passed into the constructor of AccountController. The UserManager will be responsible for all user-related operations through the UserStore implementation: insert, update, delete, add to role, read claims etc. If you navigate to AccountController.cs then you’ll see that the type of the user in the type definition is ApplicationUser. So what’s IdentityUser? If you locate the ApplicationUser class in Models/IdentityModels.cs then you’ll see that it derives from IdentityUser.

You can use the ApplicationUser class to extend the functionality of the default IdentityUser entity in EF with your custom properties.

There’s an additional important object in IdentityModels.cs: ApplicationDbContext which derives from IdentityDbContext of ApplicationUser. IdentityDbContext in turn derives from DbContext which is the standard object context in EntityFramework. Hence ApplicationDbContext is also a DbContext but it has access to the User related information that IdentityDbContext carries. We said that Users and Roles are entities so it’s just as well that ApplicationDbContext gets access to them. The constructor of the ApplicationDbContext object defines the name of the connection string, which is DefaultConnection. We saw this in the previous post but now we know where it is defined. In case you’ve renamed your connection string in web.config then you need to rename it here too.

ApplicationDbContext is also where you can add your custom DbSets so that EF can create the tables for you. We’ll go into EntityFramework in the series after this one so let’s not dive into that topic too much. It suffices to say the if you have a Customer domain then you can add a DbSet of Customer like this directly in ApplicationDbContext:

public class ApplicationDbContext : IdentityDbContext<ApplicationUser>
{
        public ApplicationDbContext()
            : base("DefaultConnection")
        {
        }

	public DbSet<Customer> Customers { get; set; }
}

If you go back to AccountController.cs you’ll notice that an ApplicationDbContext object is passed into the constructor of UserStore. Now we know that ApplicationDbContext derives from IdentityDbContext. IdentityDbContext is the EF object context which will carry out the actual DB operations of SELECT, UPDATE, DELETE etc. If you’re familiar with EF then this object context class will sound familiar.

You’ll also notice the TUser generic type argument. The actual type provided in all cases is ApplicationUser. You can define your own user type if you like but the following definitions force your implementation to implement IUser:

public class UserStore<TUser> : IUserLoginStore<TUser>, IUserClaimStore<TUser>, IUserRoleStore<TUser>, IUserPasswordStore<TUser>, IUserSecurityStampStore<TUser>, IUserStore<TUser>, IDisposable where TUser : global::Microsoft.AspNet.Identity.EntityFramework.IdentityUser

public class UserManager<TUser> : IDisposable where TUser : global::Microsoft.AspNet.Identity.IUser

I think it’s fine to have that constraint as IUser is an abstraction with minimal content:

string Id { get; }
string UserName { get; set; }

IdentityDbContext also provides the mapping between the entity classes and their DB representations. It also ensures that the tables are properly created when they are needed for the first time.

In summary we can say the Microsoft.AspNet.Identity.EntityFramework library provides the EntityFramework implementation of the abstractions in the Microsoft.AspNet.Identity.Core library. Let’s look at the concrete classes in some more detail.

First go at customisation

We’ve discussed the UserManager in some detail in the previous and this post. The Login action is represented by two methods in AccountController.cs:

[AllowAnonymous]
public ActionResult Login(string returnUrl)

[HttpPost]
[AllowAnonymous]
[ValidateAntiForgeryToken]
public async Task<ActionResult> Login(LoginViewModel model, string returnUrl)

The GET action method on top is fired when the user navigates to the Login page with the login form. The upon pressing the Log in button the POST action is invoked which accepts the LoginViewModel which in turn has 3 properties:

public string UserName { get; set; }
public string Password { get; set; }
public bool RememberMe { get; set; }

You’ll see that these are populated from the form. The POST Login method is the more interesting one as far as Identity is concerned. In the body of the method the UserManager will try to find the ApplicationUser with the FindAsync method. If the user exists, i.e. it is not null, then she is signed in and redirected to the return url. Otherwise the ModelState is invalidated.

Let’s see how we can add our custom property to the ApplicationUser object. Locate the object and add the following property:

public class ApplicationUser : IdentityUser
{
	public string FavouriteProgrammingLanguage { get; set; }
}

We’ll need to extend the corresponding view model if we want to collect this information from the user. Locate RegisterViewModel in Models/AccountViewModel.cs. It will have 3 fields: username, password and confirm password. Add a 4th one:

[Required]
[Display(Name="Favourite programming language")]
[DataType(DataType.Text)]
public string FavouriteProgrammingLanguage { get; set; }

Next open Register.cshtml in the Views/Account folder. You’ll see a form-group div where the user has to confirm the password. Just underneath add a new div:

<div class="form-group">
        @Html.LabelFor(m => m.FavouriteProgrammingLanguage, new { @class = "col-md-2 control-label" })
        <div class="col-md-10">
            @Html.TextBoxFor(m => m.FavouriteProgrammingLanguage, new { @class = "form-control" })
        </div>
</div>

Back in AccountController.cs locate the POST Register method that accepts a RegisterViewModel object. You’ll see the following bit of code after the if statement:

var user = new ApplicationUser() { UserName = model.UserName };

Extend the object as follows:

var user = new ApplicationUser() { UserName = model.UserName, FavouriteProgrammingLanguage = model.FavouriteProgrammingLanguage };

Let’s run the app and see what happens. Don’t log in with the user you created before. Instead, click on the Register link. The Register page should show the new text field. Create a new user, provide a programming language and press the Register button and.. …the yellow screen of death appears:

Yellow screen of death for code first migration

We’ve changed the entity structure but the database isn’t recreated for us on the fly of course. We’ll rectify this problem in the next post which will discuss DB migrations with EF in the User data context. We’ll revisit migrations in a greater detail in a planned series on EntityFramework which will follow this series.

The new field needs to be included elsewhere like on the Manage.cshtml view for completeness, but you can do that as an exercise.

Read the next post in this series here.

You can view the list of posts on Security and Cryptography here.

Determine the presence of an element in a sequence with LINQ C#

Say we have the following string list:

string[] bands = { "ACDC", "Queen", "Aerosmith", "Iron Maiden", "Megadeth", "Metallica", "Cream", "Oasis", "Abba", "Blur", "Chic", "Eurythmics", "Genesis", "INXS", "Midnight Oil", "Kent", "Madness", "Manic Street Preachers"
, "Noir Desir", "The Offspring", "Pink Floyd", "Rammstein", "Red Hot Chili Peppers", "Tears for Fears"
, "Deep Purple", "KISS"};

The Any operator in LINQ has two versions. The paramaterless one simply returns true if the sequence contains at least one element:

bool exists = bands.Any();
Console.WriteLine(exists);

This yields true. The more exciting version of Any is where you can specify a filter in form of an element selector lambda:

bool exists = bands.Any(b => b.Length == 4);
Console.WriteLine(exists);

This gives true as we have at least one element which consists of 4 characters. The below query returns false:

bool exists = bands.Any(b => b.EndsWith("hkhkj"));
Console.WriteLine(exists);

You can view all LINQ-related posts on this blog here.

Introduction to forms based authentication in ASP.NET MVC5 Part 2

Introduction

In the previous part of this series we looked at the absolute basics of Forms Based Authentication in MVC5. Most of what we’ve seen is familiar from MVC4.

It’s time to dive into what’s behind the scenes so that we gain a more in-depth understanding of the topic.

We’ll start with the database part: where are users stored by default and in what form? We created a user in the previous post so let’s see where it had ended up.

Demo

Open the project we started building previously.

By default if you have nothing else specified then MVC will create a database for you in the project when you created your user – we’ll see how in the next series devoted to EntityFramework. The database is not visible at first within the project. Click on the Show All Files icon in the solution explorer…:

Show all files icon in solution explorer

…and you’ll see an .mdf file appear in the App_Data folder:

Database file in App_Data folder

The exact name will differ in your case of course. Double-click that file. The contents will open in the Server Explorer:

Membership tables in MVC 5

Some of these tables might look at least vaguely familiar from the default ASP.NET Membership tables in MVC4. However, you’ll see that there are a lot fewer tables now so that data is stored in a more compact format. A short summary of each table – we’ll look at some of them in more detail later:

  • _MigrationHistory: used by EntityFramework when migrating users – migrations to be discussed in the next series
  • AspNetRoles: where the roles are stored. We have no roles defined yet so it’s empty
  • AspNetUserClaims: where the user’s claims are stored with claim type and claim value. New to claims? Start here.
  • AspNetUserLogins: used by external authentication providers, such as Twitter or Google
  • AspNetUserRoles: the many-to-many mapping table to connect users and roles
  • AspNetUsers: this is where all site users are stored with their usernames and hashed passwords

As you can see the membership tables have been streamlined a lot compared to what they looked like in previous versions. They have a lot fewer columns and as we’ll see later they are very much customisable with EntityFramework.

Right-click the AspNetUsers table and select Show Table Data. You’ll see the user you created before along with the hashed password.

Database details

Go back to the Solution explorer and open up web.config. Locate the connectionStrings section. That’s where the default database connection string is stored with that incredibly sexy and easy-to-remember name. So the identity components of MVC5 will use DefaultConnection to begin with. We can see from the connection string that a local DB will be used with no extra login and password.

You can in fact change the connection string to match the real requirements of your app of course. The SQL file name is defined by the AttachDbFilename parameter. The Initial Catalog parameter denotes the database name as it appears in the SQL management studio. Change both to AwesomeDatabase:

AttachDbFilename=|DataDirectory|\AwesomeDatabase.mdf;Initial Catalog=AwesomeDatabase;

Run the application. Now two things can happen:

  • If you continued straight from the previous post of the series then you may still be logged on – you might wonder if the app is still using the old database, but it’s not the case. The application has picked up the auth cookie available in the HTTP request. In this case press Log Off to remove that cookie.
  • Otherwise you’re not logged in and you’ll see the Register and Log in links

Try to log in, it should fail. This is because the new database doesn’t have any users in it yet and the existing users in the old database haven’t been magically transported. Stop the application, press Refresh in Solution Explorer and you’ll see the new database file:

New database created automatically

Communication with the database

We’ve seen where the database is created and how to control the connection string. Let’s go a layer up and see which components communicate with the database. Identity data is primarily managed by the Microsoft.AspNet.Identity.Core NuGet library. It is referenced by the any MVC5 app where you run forms based authentication. It contains – among others – abstractions for the following identity-related elements:

  • IUser with ID and username
  • IRole with ID and role name
  • IUserStore: interface to abstract away various basic actions around a user, such as creating, deleting, finding and updating a user
  • IUserPasswordStore which implements IUserStore: interface to abstract away password related actions such as getting and setting the hashed password of the user and determining – on top all functions of an IUserStore

There’s also a corresponding abstraction for storing the user roles and claims and they all derive from IUserStore. IUserStore is the mother interface for a lot of elements around user management in the new Identity library.

There are some concrete classes in the Identity.Core library, such as UserManager and RoleManager. You can see the UserManager in action in various methods of AccountController.cs:

var result = await UserManager.CreateAsync(user, model.Password);
IdentityResult result = await UserManager.ChangePasswordAsync(User.Identity.GetUserId(), model.OldPassword, model.NewPassword);
IdentityResult result = await UserManager.AddPasswordAsync(User.Identity.GetUserId(), model.NewPassword);

The default UserManager is set in the constructor of the AccountController object:

public AccountController()
           : this(new UserManager<ApplicationUser>(new UserStore<ApplicationUser>(new ApplicationDbContext())))
{
}

public AccountController(UserManager<ApplicationUser> userManager)
{
     UserManager = userManager;
}

public UserManager<ApplicationUser> UserManager { get; private set; }

You see that we supply a UserStore object which implements a whole range of interfaces:

  • IUserLoginStore
  • IUserClaimStore
  • IUserRoleStore
  • IUserPasswordStore
  • IUserSecurityStampStore
  • IUserStore

So the default built-in UserManager object will be able to handle a lot of aspects around user management: passwords, claims, logins etc. As a starting point the UserManager will provide all domain logic around user management, such as validation, password hashing etc.

In case you want to have your custom solution to any of these components then define your solution so that it implements the appropriate interface and then you can plug it into the UserManager class. E.g. if you want to store your users in MongoDb then implement IUserStore, define your logic there and pass it in as the IUserStore parameter to the UserManager object. It’s a good idea to implement as many sub-interfaces such as IUserClaimsStore and IUserRoleStore as possible so that your custom UserStore that you pass into UserManager will be very “clever”: it will be able to handle a lot of aspects around user management. And then when you call upon e.g. UserManager.CreateAsync then UserManager will pick up your custom solution to create a user.

However, if you’re happy with an SQL server solution governed by EntityFramework then you may consider the default setup and implementations inserted by the MVC5 template. We’ll investigate those in the next post.

You can view the list of posts on Security and Cryptography here.

Converting a sequence to a dictionary using the ToDictionary LINQ operator

Say you have a sequence of objects that you’d like to convert into a Dictionary for efficient access by key. Ideally the objects have some kind of “natural” key for the dictionary such as an ID:

public class Singer
{
	public int Id { get; set; }
	public string FirstName { get; set; }
	public string LastName { get; set; }
}

IEnumerable<Singer> singers = new List<Singer>() 
		{
			new Singer(){Id = 1, FirstName = "Freddie", LastName = "Mercury"} 
			, new Singer(){Id = 2, FirstName = "Elvis", LastName = "Presley"}
			, new Singer(){Id = 3, FirstName = "Chuck", LastName = "Berry"}
			, new Singer(){Id = 4, FirstName = "Ray", LastName = "Charles"}
			, new Singer(){Id = 5, FirstName = "David", LastName = "Bowie"}
		};

It’s easy to create a singers dictionary from this sequence:

Dictionary<int, Singer> singersDictionary = singers.ToDictionary(s => s.Id);
Console.WriteLine(singersDictionary[2].FirstName);

Basic ToDictionarty operator

You supply the key selector as the argument to the operator, which will be the key of the dictionary. The operator will throw an ArgumentException if you’re trying to add two objects with the same key. Example:

return new List<Singer>() 
		{
			new Singer(){Id = 1, FirstName = "Freddie", LastName = "Mercury"} 
			, new Singer(){Id = 1, FirstName = "Elvis", LastName = "Presley"}
			, new Singer(){Id = 3, FirstName = "Chuck", LastName = "Berry"}
			, new Singer(){Id = 4, FirstName = "Ray", LastName = "Charles"}
			, new Singer(){Id = 5, FirstName = "David", LastName = "Bowie"}
		};

There are two singers with id = 1 which will cause an exception when it’s Elvis’ turn to be inserted into the dictionary.

The ToDictionary operator has an overload which allows you to change the elements inserted into the dictionary. Say you’d like to have the id as the key but only the last name as the value. You can do like this:

Dictionary<int, string> singersDictionaryNames = 
        singers.ToDictionary(s => s.Id, si => string.Format("Last name: {0}", si.LastName));
Console.WriteLine(singersDictionaryNames[2]);

Overloaded ToDictionary operator

You can view all LINQ-related posts on this blog here.

Finding unique values using the LINQ Distinct operator

Extracting unique values from a sequence of objects in LINQ is very easy using the Distinct operator. It comes in two versions: one with a default equality comparer and another which lets you provide a custom comparer.

We’ll use the following collection for the first demo:

string[] bands = { "ACDC", "Queen", "Aerosmith", "Iron Maiden", "Megadeth", "Metallica", "Cream", "Oasis", "Abba", "Blur", "Chic", "Eurythmics", "Genesis", "INXS", "Midnight Oil", "Kent", "Madness", "Manic Street Preachers"
, "Noir Desir", "The Offspring", "Pink Floyd", "Rammstein", "Red Hot Chili Peppers", "Tears for Fears"
, "Deep Purple", "KISS"};

These are all unique values so let’s create some duplicates:

IEnumerable<string> bandsDuplicated = bands.Concat(bands);

The first prototype of Distinct will compare the string values using the standard default string comparer as all elements in the array are strings. The following bit of code will find the unique values and print them on the console:

IEnumerable<string> uniqueBands = bandsDuplicated.Distinct();
foreach (String unique in uniqueBands)
{
	Console.WriteLine(unique);
}

Distinct operator with default comparer

In case you have a sequence of custom objects then it’s a good idea to let the Distinct operator know how to define if two objects are equal. Consider the following object:

public class Singer
{
	public int Id { get; set; }
	public string FirstName { get; set; }
	public string LastName { get; set; }
}

…and the following collection:

IEnumerable<Singer> singers = new List<Singer>() 
			{
				new Singer(){Id = 1, FirstName = "Freddie", LastName = "Mercury"} 
				, new Singer(){Id = 2, FirstName = "Elvis", LastName = "Presley"}
				, new Singer(){Id = 3, FirstName = "Chuck", LastName = "Berry"}
				, new Singer(){Id = 4, FirstName = "Ray", LastName = "Charles"}
				, new Singer(){Id = 5, FirstName = "David", LastName = "Bowie"}
			};

The following comparer will compare the Singer objects by their IDs:

public class DefaultSingerComparer : IEqualityComparer<Singer>
{
	public bool Equals(Singer x, Singer y)
	{
		return x.Id == y.Id;
	}

	public int GetHashCode(Singer obj)
	{
		return obj.Id.GetHashCode();
	}
}

We create some duplicates:

IEnumerable<Singer> singersDuplicates = singers.Concat(singers);

…and then select the unique values:

IEnumerable<Singer> uniqueSingers = singersDuplicates.Distinct(new DefaultSingerComparer());
foreach (Singer singer in uniqueSingers)
{
	Console.WriteLine(singer.Id);
}

Distinct operator with custom comparer

You can view all LINQ-related posts on this blog here.

Introduction to forms based authentication in ASP.NET MVC5 Part 1

Introduction

ASP.NET MVC5 comes with a number of new elements regarding user management and security. When you create a new MVC 5 web app you’ll be able to choose between 4 default authentication types:

  • No authentication, i.e. anonymous users can access your site
  • Individual user accounts: the traditional way to log onto the site using a login form. The user store is backed by SQL identity tables. You can also enable some well known auth providers: Twitter, Google, Facebook and Microsoft where you don’t need to worry about the password
  • Organisational accounts: Active Directory Federation Services (ADFS), used within organisations that manage their users in ADFS, typically coupled with Single SignOn for the applications within the organisation. You can enable Azure-based ADFS as well
  • Windows auth: Windows Active Directory authentication for intranet apps. Your Windows login credentials will be used to access the internal applications of your company. This is somewhat like a stripped down version of the organisational accounts option

In this blog series we’ll look at the new identity features of MVC5.

Forms based authentication

Fire up Visual Studio 2013 and select the ASP.NET Web Application template in the New Project window. Give the project some name and click OK. A new window will open where you can select additional templates. Pick MVC. Press the ‘Change authentication’ button and make sure that ‘Individual user accounts’ is selected:

Individual user accounts option

Visual Studio will set up a forms-enabled MVC app for you without any extra effort. Run the web app and you’ll be directed to the default home page. You’ll see that it comes with the Register and Log in links:

Register and log in links in MVC 5 web app

If you used Forms based auth in MVC 4 then this is of course no surprise to you. Click Register and create a user. If everything goes well you’ll be automatically logged in and you’ll see your username instead of “Register”:

User registered and logged in

You can click on “Hello (your name)” to manage save your password if you want. Click Log off and then log in again with your credentials to check if it works fine. It should.

The Layout view

The top menu of the MVC 5 template is controlled by _Layout.cshtml in the Views/Shared folder. Open that file. You’ll see the links for Home, About and Contact. Below those links you’ll have a partial view called _LoginPartial. _LoginPartial is located in the same folder. Open it and let’s see what it contains.

There’s an if-else statement which tweaks what the user sees based on the Request.IsAuthenticated property. This property is set by the framework depending on whether the current user has logged on or not.

The user name is extracted using the User.Identity.GetUserName() method. User is an IPrincipal object and represents the currently logged-on user. Identity is the IIdentity belonging to the user which contains a small set of information about the user such as the user name or the authentication type. User.Identity is also set by the framework just like with the IsAuthenticated property.

You can read the User object anywhere within the controllers and views, i.e. where a valid HTTP session is available. Open Controllers/HomeController.cs and add the following code to Index() just above the return statement:

string userName = User.Identity.Name;

Set a breakpoint within Index and run the application. You’ll see that the username can be easily extracted this way.

Restrict access

There’s little point in authenticating users if you don’t limit the access to certain parts of your website to authenticated users only.

Right-click the Controllers folder and select Add, Controller. In the Add Scaffold window select the topmost option, i.e. Empty. Call it CustomersController. Add the following two methods to the controller:

public string SecretName()
{
	return "This is the secret customer name.";
}

public string PublicName()
{
	return "This is the public customer name.";
}

The goal is to protect access to the SecretName action and only let anonymous users read the public name.

Run the application and log off if you’re still logged in. Then navigate to the above actions:

  • localhost:xxxx/Customers/publicname
  • localhost:xxxx/Customers/secretname

Not surprisingly you can access both methods without having to log in first.

If your only requirement is to restrict access to a controller action to authenticated users only then you can use the Authorize attribute like this:

[Authorize]
public string SecretName()
{
	return "This is the secret customer name.";
}

Re-run the app and navigate to the secretname action. You should see that you’re redirected to the login page. Log in and you’ll see the secret. Note the URL of the Login page: Account/Login. It is defined in a Katana component in Startup.Auth.cs in the App_Start folder:

app.UseCookieAuthentication(new CookieAuthenticationOptions
{
          AuthenticationType = DefaultAuthenticationTypes.ApplicationCookie,
          LoginPath = new PathString("/Account/Login")
});

If you don’t know what OWIN and Katana mean then you can start here.

The ReturnUrl query string in the URL will store which controller and action you’ve tried to access. It will be fed to the POST Login action of AccountController:

public async Task<ActionResult> Login(LoginViewModel model, string returnUrl)

Now log off and navigate to /Customers/publicname. It should still be available to anonymous users.

What if you want to restrict access to all actions within the controller? You can apply the Authorize attribute on the controller level:

[Authorize]
public class CustomersController : Controller

Run the app again and navigate to /Customers/publicname. It is now also a restricted site so you’ll be redirected to the login page.

You can override the controller level Authorize attribute by decorated the individual action(s) with the AllowAnonymous attribute:

[AllowAnonymous]
public string PublicName()
{
	return "This is the public customer name.";
}

Run the app and verify that /Customers/publicname is publicly available again.

The Authorize attribute accepts a couple of parameters to further refine the access filter. Examples:

[Authorize(Users="andras,admin")]
[Authorize(Roles="admin,poweruser")]

You can probably guess what they mean: allow users with specific user names – andras and admin – or only allow users who have either admin and power user role to access an action.

You can test this as follows. Add the following attribute to SecretName:

[Authorize(Users = "elvis,bob")]
public string SecretName()
{
	return "This is the secret customer name.";
}

Run the app and navigate to the secret name action. Log in with your user. You should see that you’re immediately redirected back to the Login page – unless you selected ‘elvis’ or ‘bob’ as the user name in the sign up process. In that case the great secret will be revealed to you.

We’ve now seen the basics of forms based authentication in MVC5. We’ll dig much deeper in the coming blog posts.

Read the next post in this series here.

You can view the list of posts on Security and Cryptography here.

Grouping elements in LINQ .NET using GroupBy and an EqualityComparer

The GroupBy operator has the same function as GROUP BY in SQL: group elements in a sequence by a common key. The GroupBy operator comes with 8 different signatures. Each returns a sequence consisting of objects that implement the IGrouping interface of type K – the key type – and T – the type of the objects in the sequence. IGrouping implements IEnumerable of T. So when we iterate through the result the we can first look at the outer sequence of keys and then the inner sequence of each object with that key.

The simplest version of GroupBy accepts a Func delegate of T and K, which acts as the key selector. It will compare the objects in the sequence using a default comparer. E.g. if you want to group the objects by their integer IDs then you can let the default comparer do its job. Another version of GroupBy lets you supply your own comparer to define a custom grouping or if the Key is an object where you want to define your own rules for equality.

We’ll need an example sequence which has an ID. In the posts on LINQ we often take the following collections for the demos:

IEnumerable<Singer> singers = new List<Singer>() 
	{
		new Singer(){Id = 1, FirstName = "Freddie", LastName = "Mercury"} 
		, new Singer(){Id = 2, FirstName = "Elvis", LastName = "Presley"}
		, new Singer(){Id = 3, FirstName = "Chuck", LastName = "Berry"}
		, new Singer(){Id = 4, FirstName = "Ray", LastName = "Charles"}
		, new Singer(){Id = 5, FirstName = "David", LastName = "Bowie"}
	};
IEnumerable<Concert> concerts = new List<Concert>()
	{
		new Concert(){SingerId = 1, ConcertCount = 53, Year = 1979}
		, new Concert(){SingerId = 1, ConcertCount = 74, Year = 1980}
		, new Concert(){SingerId = 1, ConcertCount = 38, Year = 1981}
		, new Concert(){SingerId = 2, ConcertCount = 43, Year = 1970}
		, new Concert(){SingerId = 2, ConcertCount = 64, Year = 1968}
		, new Concert(){SingerId = 3, ConcertCount = 32, Year = 1960}
		, new Concert(){SingerId = 3, ConcertCount = 51, Year = 1961}
		, new Concert(){SingerId = 3, ConcertCount = 95, Year = 1962}
		, new Concert(){SingerId = 4, ConcertCount = 42, Year = 1950}
		, new Concert(){SingerId = 4, ConcertCount = 12, Year = 1951}
		, new Concert(){SingerId = 5, ConcertCount = 53, Year = 1983}
	};

The singers collection won’t actually be needed for the code example, it simply shows the purpose of the concerts collection. Let’s imagine that our Singers collection includes both male and female singers and that ids below 3 are female singers and the others are all male singers. Our goal is to group the concerts based on gender using this information. We can have the following custom equality comparer:

public class SingerGenderComparer : IEqualityComparer<int>
{
	private int _femaleSingerIdLimit = 3;
		
	public bool Equals(int x, int y)
	{
		return IsPerformedByFemaleSinger(x) == IsPerformedByFemaleSinger(y);
	}

	public int GetHashCode(int obj)
	{
		return IsPerformedByFemaleSinger(obj) ? 1 : 2;
	}

	public bool IsPerformedByFemaleSinger(int singerId)
	{
		return singerId < _femaleSingerIdLimit;
	}
}

Here’s the grouping of the Concerts collection using the custom comparer:

SingerGenderComparer comparer = new SingerGenderComparer();

IEnumerable<IGrouping<int, Concert>> concertGroups = concerts.GroupBy(c => c.SingerId, comparer);
foreach (IGrouping<int, Concert> concertGroup in concertGroups)
{
	Console.WriteLine("Concerts of {0} singers: ", comparer.IsPerformedByFemaleSinger(concertGroup.Key) ? "female" : "male");
	foreach (Concert concert in concertGroup)
	{
		Console.WriteLine("Number of concerts: {0}, in the year of {1} by singer {2}.", concert.ConcertCount, concert.Year, concert.SingerId);
	}
}

This yields the following output:

GroupBy output with custom comparer and default selector

You can define the type of object that will be selected from the base sequence using another version of GroupBy which allows you to provide a key selector:

IEnumerable<IGrouping<int, int>> concertGroupsFiltered = concerts.GroupBy(c => c.SingerId, c => c.ConcertCount, comparer);
foreach (IGrouping<int, int> concertGroup in concertGroupsFiltered)
{
	Console.WriteLine("Concerts of {0} singers: ", comparer.IsPerformedByFemaleSinger(concertGroup.Key) ? "female" : "male");
	foreach (int concertCount in concertGroup)
	{
		Console.WriteLine("Number of concerts: {0}.", concertCount);
	}
}

…which gives the following output:

GroupBy output with custom comparer and custom selector

You can view all LINQ-related posts on this blog here.

RabbitMQ in .NET C#: more complex error handling in the Receiver

Introduction

In the previous part on RabbitMQ .NET we looked at ways how to reject a message if there was an exception while handling the message on the Receiver’s side. The message could then be discarded or re-queued for a retry. However, the exception handling logic was very primitive in that the same message could potentially be thrown at the receiver infinitely causing a traffic jam in the messages.

This post builds upon the basics of RabbitMQ in .NET. If you are new to this topic you should check out all the previous posts listed on this page. I won’t provide any details on bits of code that we’ve gone through before.

Most of the posts on RabbitMQ on this blog are based on the work of RabbitMQ guru Michael Stephenson.

So we cannot just keep retrying forever. We can instead finally discard the message after a certain amount of retries or depending on what kind of exception was encountered.

The logic around retries must be implemented in the receiver as there’s no simple method in RabbitMQ .NET, like “BasicRetry”. Why should there be anyway? Retry strategies can be very diverse so it’s easier to let the receiver handle it.

The strategy here is to reject the message without re-queuing it. We’ll then create a new message based on the one that caused the exception and attach an integer value to it indicating the number of retries. Then depending on a maximum ceiling we either create yet another message for re-queuing or discard it altogether.

We’ll build on the demo we started on in the previous post referred to above so have it ready.

Demo

We’ll reuse the queue from the previous post which we called “BadMessageQueue”. We’ll also reuse the code in BadMessageSender as there’s no variation on the Sender side.

BadMessageReceiver will however handle the messages in a different way. Currently there’s a method called ReceiveBadMessages which is called upon from Main. Comment out that method call. Insert the following method in ReceiveBadMessages.Program.cs and call it from Main:

private static void ReceiveBadMessageExtended(IModel model)
{
	model.BasicQos(0, 1, false);
	QueueingBasicConsumer consumer = new QueueingBasicConsumer(model);
	model.BasicConsume(RabbitMqService.BadMessageBufferedQueue, false, consumer);
	string customRetryHeaderName = "number-of-retries";
	int maxNumberOfRetries = 3;
	while (true)
	{
		BasicDeliverEventArgs deliveryArguments = consumer.Queue.Dequeue() as BasicDeliverEventArgs;
		String message = Encoding.UTF8.GetString(deliveryArguments.Body);
		Console.WriteLine("Message from queue: {0}", message);
		Random random = new Random();
		int i = random.Next(0, 3);
		int retryCount = GetRetryCount(deliveryArguments.BasicProperties, customRetryHeaderName);
		if (i == 2) //no exception, accept message
		{
			Console.WriteLine("Message {0} accepted. Number of retries: {1}", message, retryCount);
			model.BasicAck(deliveryArguments.DeliveryTag, false);
		}
		else //simulate exception: accept message, but create copy and throw back
		{
			if (retryCount < maxNumberOfRetries)
			{
				Console.WriteLine("Message {0} has thrown an exception. Current number of retries: {1}", message, retryCount);
				IBasicProperties propertiesForCopy = model.CreateBasicProperties();
				IDictionary<string, object> headersCopy = CopyHeaders(deliveryArguments.BasicProperties);
				propertiesForCopy.Headers = headersCopy;
				propertiesForCopy.Headers[customRetryHeaderName] = ++retryCount;
				model.BasicPublish(deliveryArguments.Exchange, deliveryArguments.RoutingKey, propertiesForCopy, deliveryArguments.Body);
				model.BasicAck(deliveryArguments.DeliveryTag, false);
				Console.WriteLine("Message {0} thrown back at queue for retry. New retry count: {1}", message, retryCount);
			}
			else //must be rejected, cannot process
			{
				Console.WriteLine("Message {0} has reached the max number of retries. It will be rejected.", message);
				model.BasicReject(deliveryArguments.DeliveryTag, false);
			}
		}
	}
}

…where CopyHeaders and GetRetryCount look as follows:

private static IDictionary<string, object> CopyHeaders(IBasicProperties originalProperties)
{
	IDictionary<string, object> dict = new Dictionary<string, object>();
	IDictionary<string, object> headers = originalProperties.Headers;
	if (headers != null)
	{
		foreach (KeyValuePair<string, object> kvp in headers)
		{
			dict[kvp.Key] = kvp.Value;
		}
	}

	return dict;
}

private static int GetRetryCount(IBasicProperties messageProperties, string countHeader)
{
	IDictionary<string, object> headers = messageProperties.Headers;
	int count = 0;
	if (headers != null)
	{
		if (headers.ContainsKey(countHeader))
		{
			string countAsString = Convert.ToString( headers[countHeader]);
			count = Convert.ToInt32(countAsString);
		}
	}

	return count;
}

Let’s see what’s going on here. We define a custom header to store the number of retries for a message. We also set an upper limit of 3 on the number of retries. Then we accept the messages in the usual way. A random number between 0 and 3 is generated – where the upper limit is exclusive – to decide whether to simulate an exception or not. If this number is 2 then we accept and acknowledge the message, so there’s a higher probability of “throwing an exception” just to make this demo more interesting. We also extract the current number of retries using the GetRetryCount method. This helper method simply checks the headers of the message for the presence of the custom retry count header.

If we simulate an exception then we need to check if the current retry count has reached the max number of retries. If not then the exciting new stuff begins. We create a new message where we copy the elements of the original message. We also set the new value of the retry count header. We send the message copy back to where it came from and acknowledge the original message. Otherwise if the max number of retries has been reached we reject the message completely using the BasicReject method we saw in the previous part.

Run both the Sender and Receiver apps and start sending messages from the Sender. Depending on the random number generated in the Receiver you’ll see a differing number of retries but you may get something like this:

Advanced retry console output

We can see the following here:

  • Message hello was rejected at first and then accepted after 1 retry
  • Message hi was accepted immediately
  • Message bye was accepted after 2 retries
  • Message seeyou was rejected completely

So we’ve seen how to add some more logic into how to handle exceptions.

Other considerations and extensions:

  • You can specify different max retries depending on the exception type. In that case you can add the exception type to the headers as well
  • You might consider storing the retry count somewhere else than the message itself, e.g. within the Receiver – the advantage of storing the retry count in the message is that if you have multiple receivers waiting for messages from the same queue then they will all have access to the retry property
  • If there’s a dependency between messages then exception handling becomes a bigger challenge: if message B depends on message A and message A throws an exception, what do we do with message B? You can force related messages to be processed in an ordered fashion which will have a negative impact on the message throughput. On the other hand you may simply ignore this scenario if it’s not important enough for your case – “enough” depends on the cost of slower message throughput versus the cost of an exception in interdependent messages. Somewhere between these two extremes you can decide to keep the order of related messages only and let all others be delivered normally. In this case you can put the sequence number, such as “5/10” in the header so that the receiver can check if all messages have come in correctly. If you have multiple receivers then the sequence number must be stored externally so that all receivers will have access to the same information. Otherwise you can have a separate queue or even a separate RabbitMQ instance for related messages in case the proportion of related messages in total number of messages is small.

View the list of posts on Messaging here.

Converting a sequence of objects into a Lookup with LINQ C#

A Lookup in .NET is one of the lesser known data structures. It is similar to a Dictionary but the keys are not unique. You can insert multiple elements for the same key.

Say you have the following object and collection:

public class Singer
{
	public int Id { get; set; }
	public string FirstName { get; set; }
	public string LastName { get; set; }
	public int BirthYear { get; set; }
}
IEnumerable<Singer> singers = new List<Singer>() 
	{
		new Singer(){Id = 1, FirstName = "Freddie", LastName = "Mercury", BirthYear=1964}
		, new Singer(){Id = 2, FirstName = "Elvis", LastName = "Presley", BirthYear = 1954}
		, new Singer(){Id = 3, FirstName = "Chuck", LastName = "Berry", BirthYear = 1954}
		, new Singer(){Id = 4, FirstName = "Ray", LastName = "Charles", BirthYear = 1950}
		, new Singer(){Id = 5, FirstName = "David", LastName = "Bowie", BirthYear = 1964}
	};

You can group the singers into an ILookup as follows:

ILookup<int, Singer> singersByBirthYear = singers.ToLookup(s => s.BirthYear);
IEnumerable<Singer> filtered = singersByBirthYear[1964];
foreach (Singer s in filtered)
{
	Console.WriteLine(s.LastName);
}

…which outputs “Mercury” and “Bowie”.

You can also set the elements inserted into the ILookup using an overloaded variant where you specify the element selector:

ILookup<int, string> singerNamesByBirthYear = singers.ToLookup(s => s.BirthYear, si => string.Concat(si.LastName, ", ", si.FirstName));
IEnumerable<string> filtered2 = singerNamesByBirthYear[1964];
foreach (string s in filtered2)
{
	Console.WriteLine(s);
}

…which prints “Mercury, Freddie” and “Bowie, David”.

You can view all LINQ-related posts on this blog here.

Grouping elements in LINQ .NET using GroupBy

The GroupBy operator has the same function as GROUP BY in SQL: group elements in a sequence by a common key. The GroupBy operator comes with 8 different signatures. Each returns a sequence consisting of objects that implement the IGrouping interface of type K – the key – and T – the type of the objects in the sequence. IGrouping implements IEnumerable of T. So when we iterate through the result the we can first look at the outer sequence of keys and then the inner sequence of each object with that key.

The simplest version of GroupBy accepts a Func delegate of T and K, which acts as the key selector. We’ll need an example sequence which has an ID. In the posts on LINQ we often take the following collections for the demos:

IEnumerable<Singer> singers = new List<Singer>() 
	{
		new Singer(){Id = 1, FirstName = "Freddie", LastName = "Mercury"} 
		, new Singer(){Id = 2, FirstName = "Elvis", LastName = "Presley"}
		, new Singer(){Id = 3, FirstName = "Chuck", LastName = "Berry"}
		, new Singer(){Id = 4, FirstName = "Ray", LastName = "Charles"}
		, new Singer(){Id = 5, FirstName = "David", LastName = "Bowie"}
	};
IEnumerable<Concert> concerts = new List<Concert>()
	{
		new Concert(){SingerId = 1, ConcertCount = 53, Year = 1979}
		, new Concert(){SingerId = 1, ConcertCount = 74, Year = 1980}
		, new Concert(){SingerId = 1, ConcertCount = 38, Year = 1981}
		, new Concert(){SingerId = 2, ConcertCount = 43, Year = 1970}
		, new Concert(){SingerId = 2, ConcertCount = 64, Year = 1968}
		, new Concert(){SingerId = 3, ConcertCount = 32, Year = 1960}
		, new Concert(){SingerId = 3, ConcertCount = 51, Year = 1961}
		, new Concert(){SingerId = 3, ConcertCount = 95, Year = 1962}
		, new Concert(){SingerId = 4, ConcertCount = 42, Year = 1950}
		, new Concert(){SingerId = 4, ConcertCount = 12, Year = 1951}
		, new Concert(){SingerId = 5, ConcertCount = 53, Year = 1983}
	};

The singers collection won’t actually be needed for the code example, it simply shows the purpose of the concerts collection. Here’s the grouping of the Concerts collection by singer ID:

IEnumerable<IGrouping<int, Concert>> concertGroups = concerts.GroupBy(c => c.SingerId);
foreach (IGrouping<int, Concert> concertGroup in concertGroups)
{
	Console.WriteLine("Concerts for singer of ID {0}:", concertGroup.Key);
	foreach (Concert concert in concertGroup)
	{
		Console.WriteLine("Number of concerts: {0}, in the year of {1}.", concert.ConcertCount, concert.Year);
	}
}

GroupBy operator basic output

You can define the type of object that will be selected from the base sequence using another version of GroupBy which allows you to provide a key selector:

IEnumerable<IGrouping<int, int>> concertGroupsFiltered = concerts.GroupBy(c => c.SingerId, c => c.ConcertCount);
foreach (IGrouping<int, int> concertGroup in concertGroupsFiltered)
{
	Console.WriteLine("Concerts for singer of ID {0}:", concertGroup.Key);
	foreach (int concertCount in concertGroup)
	{
		Console.WriteLine("Number of concerts: {0}.", concertCount);
	}
}

GroupBy operator key selector output

View the list of posts on LINQ here.

Elliot Balynn's Blog

A directory of wonderful thoughts

Software Engineering

Web development

Disparate Opinions

Various tidbits

chsakell's Blog

WEB APPLICATION DEVELOPMENT TUTORIALS WITH OPEN-SOURCE PROJECTS

Once Upon a Camayoc

ARCHIVED: Bite-size insight on Cyber Security for the not too technical.