MongoDB in .NET part 3: starting with POCO documents

Introduction

In the previous post we set up our MongoDb server and saw how to establish a connection to it. We’ll now start inserting objects into the database. In other words we’ll create MongoDb documents.

Note that there are basically two ways to create documents: either via low-level BSON objects or your “proper” custom POCO ones that you create in your domain layer. In this tutorial we’ll not cover BSON documents. I think most developers prefer the more strongly typed POCO way of working with documents. With BSON you have to declare your object structure through strings.

Here’s a code example of creating a BSON document in C# from the C# driver tutorial:

BsonDocument document = new BsonDocument 
{
    { "name", name },
    { "city", city }, // not added if city is null
    { "dob", dob, dobAvailable } // not added if dobAvailable is false
};

Another example:

BsonDocument nested = new BsonDocument {
{ "name", "John Doe" },
    { "address", new BsonDocument {
        { "street", "123 Main St." },
        { "city", "Centerville" },
        { "state", "PA" },
        { "zip", 12345}
    }}
};

We’ll only look at POCOs here. I think you’ll agree that it’s easier to work with “real” objects for all CRUD operations and queries.

Data serialisation

The C# driver will translate to and from BSON and POCO automatically through data (de)serialisation. The default serialisation is very simple. Say we have the following POCO:

public class Customer
{
	public string Name { get; set; }
	public string Address { get; set; }
}

The default JSON representation of this POCO will be:

{
    "Name" : "Elvis"
    , "Address" : "Graceland"
}

The POCO will be saved in the BSON format but the above JSON helps visualise the binary source. If the POCO has some sequence of objects, e.g.:

public class Customer
{
	public string Name { get; set; }
	public string Address { get; set; }
        IEnumerable<string> Telephones { get; set; }
}

The it’s translated into a JSON array:

{
    "Name" : "Elvis"
    , "Address" : "Neverland"
    , "Telephones" : ["123", "456"]
}

Nested objects such as…

public class Customer
{
	public string Name { get; set; }
	public string Address { get; set; }
	IEnumerable<string> Telephones { get; set; }
	public WebPage PublicPage { get; set; }
}

public class WebPage
{
	public bool IsSsl { get; set; }
	public string Domain { get; set; }
}

…are represented as nested documents like here:

{
    "Name" : "Elvis"
    , "Address" : "Neverland"
    , "Telephones" : ["123", "456"]
    , "PublicPage" : { "IsSsl": true, "Domain" : "company.com" }
}

I think you’re beginning to see the modelling advantage of MongoDb compared to RMDBS databases. You can have any kind of object structure, the resulting document will immediately store it in the appropriate JSON/BSON format without any extra mapping, schema changes, data migrations etc. Feel free to describe your business model in POCOs with no constraints laid by the database. Have nested objects, lists, arrays etc. inside your domain objects – MongoDb documents will be able to handle them with no extra effort.

Changing the default serialisation

Occasionally you may want to deviate from the standard serialisation mechanism: you can ignore certain properties or define a different property name. Serialisation attributes in MongoDb will help you a great deal here. They are similar to the DataMember, IgnoreMember etc. attributes in .NET serialisation. Examples:

To ignore a property use BsonIgnore:

public class Customer
{
	public string Name { get; set; }
	public string Address { get; set; }
	IEnumerable<string> Telephones { get; set; }
	[BsonIgnore]
	public WebPage PublicPage { get; set; }
}

The PublicPage property will not be serialised in the BSON document.

To store a property under a different name use the BsonElement attribute:

public class Customer
{		
	public string Name { get; set; }
	public string Address { get; set; }
	IEnumerable<string> Telephones { get; set; }
	[BsonElement("PublicWebPage")]
	public WebPage PublicPage { get; set; }
}

You might want to do this if the Customer object had a property called PublicWebPage before but was renamed to PublicPage. You still want all the old records to be deserialised so you can declare the name of the property like that.

Ignore NULL values:

public class Customer
{		
	public string Name { get; set; }
	[BsonIgnoreIfNull]
	public string Address { get; set; }
}

By default private fields are not serialised. You can use the empty BsonElement to include it in the serialisation:

public class Customer
{		
	public string Name { get; set; }
	public string Address { get; set; }
	IEnumerable<string> Telephones { get; set; }
	[BsonElement]
	private WebPage PublicPage { get; set; }
}

Declare how the property will be represented in BSON, i.e. under which type, using the BsonRepresentation attribute:

public class Customer
{
	[BsonRepresentation(MongoDB.Bson.BsonType.Double)]
	public decimal TotalOrders { get; set; }
}

The .NET decimal type has no equivalent in BSON, it’s treated as a string which makes sorting and comparing inefficient. It’s better to cast it to a BSON double which is close enough.

We saw that the unique ID field is called “_id” by default and is of type ObjectId. You can declare the ID field using the BsonId attribute:

public class Customer
{
	[BsonId]
	public int CustomerId { get; set; }
}

The ID will still be serialised as “_id” in the document but you can refer to it as “CustomerId” in the POCO. Keep in mind, that it’s your responsibility to create unique integer IDs – there’s no equivalent of SQL Server’s auto-increment (1,1) in MongoDb. In case you stick to the ObjectId type then MongoDb will create unique values for you.

Dates are always a different beast with all the time zones and formats. MongoDb translates all local dates to UTC automatically when a date field is serialised. However, it can create problems when the date is deserialised. You’ll by default get the UTC date whereas you may expect the local one. I think it’s always wise to work only with UTC in your domains and convert dates for views only, but you can use the BsonDateTimeOptions attribute to override the default behaviour:

public class Customer
{
	[BsonDateTimeOptions(Kind = DateTimeKind.Local)]
	public DateTime CustomerSince { get; set; }
}

This way the UTC date stored in the document will be converted to the local time zone of the computer.

In case you only want to store the “date” part of a date, i.e. without the hours:minutes:seconds bit then use the DateOnly parameter of BsonDateTimeOptions:

public class Customer
{
	[BsonDateTimeOptions(Kind = DateTimeKind.Local, DateOnly = true)]		
	public DateTime CustomerSince { get; set; }
}

If you remove a property from your POCO then the document will adjust automatically during serialisation as we said before. However, the C# driver won’t like it when deserialising the BSON representation. It will throw a FileFormatException in case a property in the BSON document doesn’t match any of the properties in the POCO. You can make the driver ignore unmapped properties using the BsonIgnoreExtraElements class-level attribute:

[BsonIgnoreExtraElements]
public class Customer
{
	[BsonDateTimeOptions(Kind = DateTimeKind.Local, DateOnly = true)]		
	public DateTime CustomerSince { get; set; }
}

In the next post we’ll start building the POCOs for our demo car rental application.

View the posts related to data storage here.

MongoDB in .NET part 2: setup

Introduction

In the previous post we covered the basics of MongoDb. In this post we’ll look at how to set it up in Windows and how to connect to it through the C# driver.

Note that the C# driver is actually a .NET driver which can be called from Visual Basic as well.

Setup on Windows

Let’s set up MongoDb on the localhost. Navigate to the MongoDb downloads page and click the Windows download link:

This will download an msi installer. Step through the installer. Choose the “typical” installation mode when the question comes. The installer will put the MongoDb files in the Program Files/MongoDB x.x Standard folder:

The installation package is self-contained, i.e. there’s no dependency on other elements. We’ll put it somewhere where it’s easier to find. Create a folder called mongodb on your C drive and cut/paste the contents of the Program Files/MongoDB x.x Standard into it. MongoDb will by default use a folder called “data” on your C drive to store the documents. So insert a folder called “data” on c:\. Within the “data” folder create another folder called “db”.

Let’s first connect to MongoDb manually. Open a command prompt, navigate to c:\mongodb\bin and issue the following command:

mongod.exe

This will start the MongoDb database engine. You’ll see that it’s waiting for connections:

Open up another command prompt, navigate to c:\mongodb\bin and start the client:

mongo.exe

So we managed to connect to the MongoDb engine from a client. You can write direct JavaScript commands in the client to create databases, collections, insert documents etc. I will not go through these low level operations here – I’m only familiar with the basics myself anyway. We’ll do all that in C#. However, if you’re interested in those direct commands then you can check out the official manual.

Close both command prompts to start afresh and open a new one. Make sure it has administrator rights. It’s not convenient having to start the DB engine all the time we want to talk to it. Fortunately MongoDb can be set up as a Windows service. We’ll need a folder for the log files so add a folder called “log” to c:\mongodb. Create a configuration file for the logpath option for MongoDb in the command prompt by issuing this command:

echo logpath=C:\mongodb\log\mongo.log> C:\mongodb\mongod.cfg

This will create a mongod.cfg file in c:\mongodb.

Next run the following command:

C:\mongodb\bin\mongod.exe –config C:\mongodb\mongod.cfg –install

This will create the MongoDB Windows service. Note that it is a double-dash in front of “config” and “install”, this is not easily visible due to the rendering of this post on your screen. Let’s start the service:

net start mongodb

The command prompt should say that the service was started successfully. If you get an error message that the service doesn’t exist then re-run the installation command from an Administrator command prompt.

Let’s connect to MongoDb again from the client:

c:\mongodb\bin\mongo.exe

It should say “connecting to: test” so we’ve connected to the MongoDb engine that is now up and running thanks to the Windows service. “Test” is the default database that the mongo client connects to first.

Connecting to MongoDb in code

We’ll build up a simple MVC web application where people can rent cars.

Warning: I’ll be concentrating on Mongo-related functionality and ignore software practices like SOLID or layered architecture. If you are familiar with those concepts then you’ll see a lot of “big no-no” code here. However, I want to concentrate on MongoDb. It’s your responsibility to organise your code to fit good software engineering design.

Open Visual Studio 2012/2013 and create a new ASP.NET MVC web application called CarRentalWeb. It doesn’t matter if it’s MVC4 or 5 really, we’ll be concentrating on MongoDb. I’ll build this demo using MVC 4 in VS 2012 but MVC 5 in VS 2013 should be equally fine. If you have VS 2013 then when you create the project make sure to select the “No authentication” auth type. We don’t want to be sidetracked by security issues. Add the following NuGet package to the references list:

A general note: the MongoDb C# driver evolves quite quickly with lots of additions and deprecated code. At the time of writing this post the most recent version was 1.9.1. By the time you’re reading this there may well be a new version. It’s happened before to me that an updated release included code that immediately broke the existing code I wrote. It is possible that the code I’m going to write in this demo gets outdated quickly but hopefully it’ll be enough for you to follow along.

The NuGet package will install the following libraries:

Just like an SQL Server driver such as EntityFramework the MongoDb driver will also need a connection string. There’s a whole list of examples on how to construct MongoDb connection strings here. Open web.config and add the following connection string to the connectionStrings section:

<add name="CarRentalConnectionString" connectionString="mongodb://localhost"/>

This is probably as easy as it gets.

Open HomeController.cs in the Controllers folder and add the following private fields and a constructor to it:

private MongoClient _mongoClient;
private MongoServer _mongoServer;
private MongoDatabase _mongoDatabase;

public HomeController()
{
	_mongoClient = new MongoClient(ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString);
	_mongoServer = _mongoClient.GetServer();
	_mongoDatabase = _mongoServer.GetDatabase(ConfigurationManager.AppSettings["CarRentalDatabaseName"]);
}

The MongoServer object will be used to connect to a database. MongoClient serves for sending commands to the server. We’ll store the name of the database in web.config. Add the following setting to the appSettings section:

<add key="CarRentalDatabaseName" value="CarRentalDatabase"/>

We haven’t created that database anywhere but we don’t need to worry. It will be created for us automatically when we insert the first document. Let’s verify that the connection can be made through reading the server build info. Erase the contents of Index() and add the following instead:

_mongoDatabase.GetCollectionNames();
return Json(_mongoDatabase.Server.BuildInfo, JsonRequestBehavior.AllowGet);

The first row is only needed to trigger a connection in the driver, it doesn’t have any other purpose. OK, let’s now run the application. If everything went well then you’ll get some long JSON on your screen:

Note the MongoDb server version: 2.6.1. Also, the C# driver we installed above was of version 1.9. All demos and code examples will be built on this configuration. If you have an older version of whichever then you might not get the same behaviour, e.g. you may see that an older version of the driver doesn’t have certain methods. Therefore make sure you have those versions are higher.

A short aside: MongoDb settings

We saw above the simplest way to declare our connection to the MongoDb server. There’s however a rich imperative way of declaring your MongoDb settings through the MongoClientSettings object. Examples:

MongoClientSettings settings = new MongoClientSettings();
String mongoHost = ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString);
MongoServerAddress address = new MongoServerAddress(mongoHost);
settings.ConnectTimeout = new TimeSpan(0, 1, 5);
settings.SocketTimeout = new TimeSpan(0, 1, 5);
settings.Server = address;

The above code sets the connection string and the connection timeouts. You can also set the connection pool and wait queue size with a wait queue timeout:

settings.MaxConnectionPoolSize = 1500;
settings.WaitQueueSize = 1500;
settings.WaitQueueTimeout = new TimeSpan(0, 1, 0);

It’s very likely that our database will require authentication. The following method can help you set up the MongoDb credentials list:

private static List<MongoCredential> GetMongoCredentials()
{
	string username = ConfigurationManager.AppSettings["MongoDbUserName"];
	string password = ConfigurationManager.AppSettings["MongoDbPassword"];
	string mongoDbAuthMechanism = ConfigurationManager.AppSettings["MongoDbAuthMechanism"];
	MongoInternalIdentity internalIdentity = new MongoInternalIdentity(LtpMongoDbDefaultDatabase, username);
	PasswordEvidence passwordEvidence = new PasswordEvidence(password);
	MongoCredential mongoCredential = new MongoCredential(mongoDbAuthMechanism, internalIdentity, passwordEvidence);
	List<MongoCredential> credentials = new List<MongoCredential>() { mongoCredential };
	return credentials;
}

…where “MongoDbAuthMechanism” has the following value: “MONGODB-CR”, more about that here. You can then set the credentials as follows:

settings.Credentials = GetMongoCredentials();

In case you have a replica set in place, i.e. a primary server with one ore more secondary servers then you can set the read preference and connection mode as follows:

settings.ReadPreference = new ReadPreference(ReadPreferenceMode.SecondaryPreferred);
settings.ConnectionMode = ConnectionMode.ReplicaSet;

And finally we construct the MongoServer object as follows:

MongoClient client = new MongoClient(settings);
MongoServer mongoServer = client.GetServer();
_mongoServer = mongoServer;	

The MongoClientSettings object has more properties than shown here. If you need to set some specific property to connect to your database then check out this object, it will probably have something for you.

The MongoDb context class

If you’ve used Linq to SQL or EntityFramework then you’ll know that they each have their DB context classes. There’s no equivalent in MongoDb but it can be emulated to a certain extent. We’ll wrap the code which constructs the MongoDatabase class in its own container class. Insert a new folder called MongoDb to the project. Add a class called CarRentalContext to it:

private MongoClient _mongoClient;
private MongoServer _mongoServer;
private MongoDatabase _mongoDatabase;

public CarRentalContext()
{
	_mongoClient = new MongoClient(ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString);
	_mongoServer = _mongoClient.GetServer();
	_mongoDatabase = _mongoServer.GetDatabase(ConfigurationManager.AppSettings["CarRentalDatabaseName"]);
}

public MongoDatabase CarRentalDatabase
{
	get
	{
		return _mongoDatabase;
	}
}

This way the context class can be accessed from other controllers as well. Keep in mind that this context class is not nearly as sophisticated as the DB context classes of Linq to SQL or EntityFramework. There’s no SubmitChanges/SaveChanges method to carry out the changes stored in the context. There’s no InsertOnSubmit, DeleteOnSubmit etc. methods either. The changes will be immediate unless you write your own Unit of Work wrapper around this simple context class. Back in HomeController erase the private fields and delete the contents of the constructor. Update the code as follows:

private CarRentalContext _carRentalContext;

public HomeController()
{
	_carRentalContext = new CarRentalContext();
}

public ActionResult Index()
{
	_carRentalContext.CarRentalDatabase.GetCollectionNames();
	return Json(_carRentalContext.CarRentalDatabase.Server.BuildInfo, JsonRequestBehavior.AllowGet);
}

I can’t stress this enough: you should never build real-life controllers like that. Make sure you understand SOLID which tells you how to inject such dependencies correctly. Re-run the application to make sure it’s still working. It should.

You’ll notice that there’s no code to connect to and disconnect from the database. The driver will handle all that for you through the connection pool. Also, MongoServer instances are cached for a given set of settings so don’t worry about using this context class multiple times. The driver will provide the cached version. Also, the 3 key objects we saw above, i.e. MongoClient, MongoServer and MongoDatabase are all threadsafe, so don’t worry about locks and data races.

In the next post we’ll be looking into (de)serialisation of POCO objects in MongoDb.

You can view all posts related to data storage on this blog here.

MongoDB in .NET part 1: foundations

Introduction

There has been a strong increase in the usage of document based NoSql databases in the previous years. One prominent example is MongoDb. The default choice for storing data in a .NET project has most often been SQL Server. While SQL Server is probably still the most popular choice .NET developers can choose from other well-tested alternatives depending on their project needs. There is a whole suite of NoSQL databases out there besides MongoDb. Some examples:

• Redis
• CouchDb
• Neo4j, a graph database
• riak
• RavenDb

At the time of writing this post MongoDb was the most popular NoSQL database according to db-engines. Even if you’re a staunch relational DBMS advocate and think you’ll never use anything else for your data storage solution it’s still beneficial to know that there are other tools out there. Also, you can have have a mixed data storage strategy where you store some objects in a relational DBMS and some others in a document store depending on the objects’ nature and the advantages and disadvantages of each storage mechanism.

You know that “we’ve always done it this way” is a dangerous mindset. Studying MongoDb can widen your skills set and offer a different point of view to solving data storage challenges.

In this series we’ll explore MongoDb in more details and how it can used in a .NET project.

MongoDb terms

As hinted at above MongoDb is a document-based database. In SQL Server the records are stored in rows where each column usually represents a property of the object that the table describes. In MongoDb however, information is stored in JSON documents – or to be exact, in Binary JSON, i.e. BSON. Here’s an example from the MongoDb homepage:

{
"_id": 1,
"name": {
"first": "John",
"last": "Backus"
},
"contribs": ["Fortran",
"ALGOL",
"Backus-Naur Form",
"FP"],
"awards": [{
"award": "W.W. McDowell Award",
"year": 1967,
"by": "IEEE Computer Society"
},
{
"award": "Draper Prize",
"year": 1993,
"by": "National Academy of Engineering"
}]
}

You’ll immediately notice that the properties such as “awards” can be arrays where each element in turn has their own properties. If you’re familiar with JSON then this is no surprise to you. As the documents are stored in binary format you cannot just open a BSON document in a text editor however.

The above JSON/BSON string is an example of a document. It consists of a set of key-value pairs such as _id = 1. Each key-value pair is an element or a field. You can see that “name” itself is a document within the larger document, also called an embedded document. A group of related documents is called a collection. A collection is similar to a table in SQL Server.

Each document has a unique ID field denoted by “_id”. It has the same function as a unique primary key in an SQL Server table. It can be of any type really but it’s customary to make it of type ObjectId which looks similar to a GUID. No matter which type you use, the ID must be unique across the collection of documents.

Advantages

There are some key advantages of MongoDb compared to traditional “table-based” databases:

• Dynamic data structure with flexible schemas: you don’t need to define columns and tables. You can in fact store pretty much anything within the same collection
• Due to the lack of strict schemas data migrations become a lot easier too: if you change your domain structure, i.e. you business objects in code, the document will store the objects correspondingly. You can force a change in the schema through changing your custom objects automatically
• You don’t need separate tables to show relationships: the above JSON structure shows you that there’s no need for a separate “awards” collection, you can show the relationship directly within the document – there’s no need for foreign keys and constraints. If you extract a single item from a collection then you’ll immediately get its associated objects: order with all order items, rock band with all concerts, making it a breeze to perform operations on those linked objects
• MongoDb documents therefore allow storing your objects in an object oriented fashion which is sometimes difficult and awkward to solve in SQL Server with separate tables and keys
• Due to the lack of constraints such as secondary keys updating and deleting items will be easier: there’s no cascade delete, no orphans
• Speed: MongoDb is very fast and efficient in querying and inserting items in a collection

Disadvantages

All of the above is very well and good but there are couple of things that you need to be aware of if you’re coming from an SQL Server environment – which probably at least 95% of .NET developers do.

• Lack of professional tools: with SQL Server you can use SSMS for some very advanced GUI-based database operations, such as database profiling, SQL jobs, a query editor, IntelliSense and a whole lot more. There’s no equivalent in MongoDb. If you want to work directly with collections without one of the drivers – C#, Java, Php etc – then you’ll need to write your code in a console window or develop a custom solution yourself. There’s one visual tool though that you can use to improve the administration of your MongoDb database: RoboMongo
• As of writing this post MongoDb doesn’t support transactions
• Many developers will say that the lack of a schema is actually a disadvantage: you cannot associate objects through keys, you cannot force a compulsory data structure like “NOT NULL”
• No stored procedures
• Difficult to retrieve lost data

You can read more about the advantages and disadvantages of MongoDb here.

Drivers

While you can interact with MongoDb directly in a command window you’ll most certainly prefer to do that in code through a client library, a.k.a a driver. There are drivers available for most mainstream languages, like C#, Java or PHP. You can check out the full list of language-specific drivers here.

In the next post we’ll look at how to set up MongoDb and how to connect to it from code.

You can view all posts related to data storage on this blog here.