MongoDB in .NET part 10: other file operations

Introduction

In the previous post on MongoDb we talked about inserting files to GridFS and linking them to a Car object. In this post we’ll look at how to read, delete and update a file.

We’ll build on the CarRental demo project we’ve been working on, so have it ready in Visual Studio.

Reading a file

The primary way of reading a file with the C# driver of MongoDb is the Download method of MongoGridFS and its numerous overloads. With Download you can extract the contents of a file to a stream or to the local file system. With the MongoGridFS.Open method you can put the contents of a file to a Stream but you can only locate the file by file name. There are also the MongoGridFS.Find methods – Find, FindOne, FindAll, FindOneById which allows you to find metadata on a file and indirectly open the contents of the file through the MongoGridFSFileInfo object that the Find methods return. MongoGridFSFileInfo has methods to open a file: Open, OpenRead and OpenText. The Find method lets you search for a specific file using an IMongoQuery object. These Find methods are very much the same as what we saw in the post on querying documents.

Deleting and updating a file

Deletes and updates are handled under the same section as there’s no separate update method. An update means first removing a file and then inserting a new one instead.

Demo

Let’s show the image associated with the Car object on Image.cshtml if one exists. We’ll need a helper method on the CarViewModel object to determine whether it has an image. Add the following property to CarViewModel.cs:

public bool HasImage
{
	get
	{
		return !string.IsNullOrEmpty(ImageId);
	}
}

We’ll retrieve the Image from a controller action. Add a new Controller to the Controllers folder called ImagesController. Select the Empty MVC Controller template type. Make it derive from BaseController and insert an Image action method:

public class ImagesController : BaseController
{        
        public ActionResult Image(string imageId)
        {
		MongoGridFSFileInfo imageFileInfo = CarRentalContext.CarRentalDatabase.GridFS.FindOneById(new ObjectId(imageId));
		return File(imageFileInfo.OpenRead(), imageFileInfo.ContentType);
        }
}

The last missing piece is to extend Image.cshtml to show the image. Add the following markup just below the closing brace of the Html.BeginForm statement:

@if (Model.HasImage)
{
	<img src="@Url.Action("Image", "Images", new { imageId = @Model.ImageId})" />
}

Run the application, navigate to /cars and click on the Image link of a Car which has a valid image file. If everything’s gone fine then you should see the associated image:

Show car image from GridFS

In case you’re wondering: that’s right, I uploaded the Windows logo from the background of my computer as the car image. It doesn’t make any difference what image you’ve associated with the Car, the main thing is that it’s shown correctly.

Deleting and updating a file

As hinted above, there’s no separate update function for files. Therefore files are updated in two steps: delete the existing one and insert a new one instead. We’ve already seen how to insert a file and link it to an object, so we only need to consider deletions in this section.

Deletions are performed with the Delete function and its overloads: delete by a query, delete by file name and delete by id. You can also delete a file by its MongoGridFSFileInfo wrapper, an instance of which we’ve seen above.

If you use the Delete(IMongoQuery) overload then all files will be deleted one by one that match the search criteria, i.e. the matching files are not deleted in an atomic operation.

We’ll extend our demo app as follows: when an image is uploaded then we check if the Car already has one. If so, then the image is replaced. Otherwise we’ll upload the file like we saw in the previous post.

Insert the following method to CarsController:

private void DeleteCarImage(Car car)
{
	CarRentalContext.CarRentalDatabase.GridFS.DeleteById(car.ImageId);
	car.ImageId = string.Empty;
	CarRentalContext.Cars.Save(car);
}

We first delete the image by its ID. Then we need to set the ImageID property of the car to an empty string as there’s no automatic mechanism that can determine that the image of a Car has been deleted and update the ImageId property. Lastly we save the Car object.

We can call this function in an extended version of POST Image:

[HttpPost]
public ActionResult Image(string id, HttpPostedFileBase file)
{
	Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
	if (!string.IsNullOrEmpty(car.ImageId))
	{
		DeleteCarImage(car);
	}
	AttachImageToCar(file, car);
	return RedirectToAction("Index");
}

That’s it. Run the app as usual and try to replace an image attached to a Car, it should go fine.

Other interesting operations

The MongoGridFS and MongoGridFSFileInfo objects have a MoveTo function. They don’t actually move anything, they set the file name metadata field to a different value. If you want to duplicate a file, then call the CopyTo method that’s available on each object.

You can also modify the metadata of a file through the SetMetada method of MongoGridFS.

The end

This was the last post in the series on MongoDb in .NET. There’s of course loads more to look at but we’d need a book to cover all aspects. However, this should be enough for you to start coding and explore MongoDb in further detail on your own.

View the posts related to data storage here.

MongoDB in .NET part 9: storing files

Introduction

In the previous post we looked at a couple of query techniques in the MongoDb C# driver. Before that in this series we saw how to store, update, delete and query “usual” objects like Car. Storing files, such as a PDF or Word file, on the other hand is a different matter. They take up a lot of space in the data store and require special data types to store the bytes.

In this post we’ll look at how files can be stored in MongoDb using its GridFS technology. GridFS stores files in MongoDb documents just like normal objects. However, it stores the file contents in chunks of 256KB. Each chunk is stored in a MongoDb document. Recall that a single MongoDocument can store 16MB of data. Below 16MB you can still opt for storing the byte array contents of a file in a single MongoDb document but for consistency you probably should store all your files in GridFS. GridFS also stores the file metadata in a separate MongoDocument with appropriate links to the constituent chunks.

GridFS is similar to a file system such as the Windows file directory. However, it is independent of the platform it’s running on, i.e. GridFS is not constrained by any limitations of the file system of the OS.

Also, as each chunk is stored in a MongoDb document, GridFS prepares the way for content replication in a Replica Set scenario.

Atomic operations are not available for files in GridFS. You cannot locate and update a file in a single step. Also, the only way to update a file is by replacing an existing one. You cannot send an Update document to the Mongo server to update a small part of a file. So GridFS is probably a good option for files that don’t change too often.

GridFS in the C# driver

GridFS is represented by the GridFS property of MongoDatabase. The GridFS instance with the default settings can be reached in our demo CarRental demo as follows:

MongoGridFS gridFsDefault = CarRentalContext.CarRentalDatabase.GridFS;

You can use the GetGridFS method of MongoDatabase to override the default settings:

MongoGridFSSettings gridFsSettings = new MongoGridFSSettings();
gridFsSettings.ChunkSize = 1024;
MongoGridFS gridFsCustom = CarRentalContext.CarRentalDatabase.GetGridFS(gridFsSettings);

The files can be retrieved using the Files property of MongoGridFS. By default it returns a collection of BsonDocuments but it can be serialised into MongoGridFSFileInfo objects. They allow us to easily extract the metadata about a file in an object oriented way: file name, size in bytes, date uploaded, content type etc. As the files stored in GridFS are independent of the local file system many of these properties can be provided by the user, such as the file name or a list of aliases.

If you ever wish to view individual chunks of a file then you can retrieve them using the Chunks property:

MongoCollection<BsonDocument> chunks = gridFsDefault.Chunks;

These cannot be serialised into any strongly types object yet. Each BsonDocument has a files_id field by which you can identify the chunks belonging to a single file. You can get the file id from the MongoGridFSFileInfo object. You can put the chunks into the right order using the numeric ‘n’ field which denotes the place of a chunk in the sequence. The binary data can be extracted using the ‘data’ field.

We can upload files using one of the overloaded Upload functions of MongoGridFS. One such overload allows us to specify a Stream, a remote file name – to change the file name representation in the metadata – and a MongoGridFSCreateOptions object. This last parameter will let us further define the options for uploading a single file: a list of aliases, chunk size, content type, custom metadata, upload date and an id. The Upload function returns a MongoGridFSFileInfo object.

Another way to upload files is to use the Create and CreateText methods of MongoDatabase.

Demo

We’ll extend our CarRental demo. The first goal is to be able to upload an image linked to a Car. Locate Index.cshtml in the Views/Cars folder. It contains a set of links for each Car entry: Edit, Details and Delete. Add the Image action link to that list:

<td>
        @Html.ActionLink("Edit", "Edit", new { id=item.Id }) |
        @Html.ActionLink("Details", "Details", new { id=item.Id }) |
        @Html.ActionLink("Delete", "Delete", new { id=item.Id }) |
       	@Html.ActionLink("Image", "Image", new { id=item.Id })
</td>

Add the following method to CarsController.cs:

public ActionResult Image(string id)
{
	Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
	return View(car.ConvertToViewModel());
}

Right-click “Image” and select Add View…:

Add car rental image view

You’ll get an empty cshtml file. We’ll fill it out in a little bit. We’ll certainly need a POST action in CarsController that accepts the posted file and the Car id. Also, we’ll need a property on the Car object to store the image id. OK, let’s start from the bottom. Add the following property to Car:

public string ImageId { get; set; }

Add it to CarViewModel.cs as well:

[Display(Name = "Image ID")]
public string ImageId { get; set; }

We won’t yet see the image itself until the next post so we’ll only show the image id first. Locate ConvertToViewModel in DomainExtensions.cs and add the new property to the conversion:

CarViewModel carViewModel = new CarViewModel()
{
	Id = carDomain.Id
	, DailyRentalFee = carDomain.DailyRentalFee
	, Make = carDomain.Make
	, NumberOfDoors = carDomain.NumberOfDoors
	, ImageId = carDomain.ImageId
};

In Views/Cars/Index.cshtml we’ll extend the table to view the image IDs:

...
<th>
	@Html.DisplayNameFor(model => model.ImageId)
</th>
...

<td>
	@Html.DisplayFor(modelItem => item.ImageId)
</td>

...

Run the application to test if it’s still working. The image IDs will be empty of course in the Cars table. Click the Image link to see if it leads us to the empty Image view.

In CarsController add the following private method:

private void AttachImageToCar(HttpPostedFileBase file, Car car)
{
	ObjectId imageId = ObjectId.GenerateNewId();
	car.ImageId = imageId.ToString();
	CarRentalContext.Cars.Save(car);
	MongoGridFSCreateOptions createOptions = new MongoGridFSCreateOptions()
	{
		Id = imageId
		, ContentType = file.ContentType
	};
	CarRentalContext.CarRentalDatabase.GridFS.Upload(file.InputStream, file.FileName, createOptions);
}

We construct a new object id and set it as the ImageId property of the Car object. We then call the Save function to update the Car in the database. Then we use an overload of the Upload function to upload the file to GridFS. Note how we got hold of the default GridFS instance through the GridFS property. As we specify the ID and the content type ourselves we can use the MongoGridFSCreateOptions to convey the values. Call this function from the following POST method in CarsController:

[HttpPost]
public ActionResult Image(string id, HttpPostedFileBase file)
{
	Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
	AttachImageToCar(file, car);
	return RedirectToAction("Index");
}

We can now fill in Image.cshtml:

@model CarRentalWeb.ViewModels.CarViewModel

@{
    ViewBag.Title = "Image";
}

<h2>Car image</h2>

@using (Html.BeginForm(null, null, FormMethod.Post, new { enctype = "multipart/form-data" }))
{
	@Html.AntiForgeryToken()

	<div>
        
		<div>
			<label>Make: </label>
			<div>
				@Model.Make
			</div>
		</div>
		
		<div>
			<label>Image: </label>
			<div>
				<input type="file" id="file" name="file"/>
			</div>
		</div>

		<div>
			<div>
				<input type="submit" value="Save"/>
			</div>
		</div>
	</div>
}

<div>
	@Html.ActionLink("Back to Cars", "Index")
</div>

Specifying multipart form data in “enctype = multipart/form-data” will enable us to post the entire form data along with the posted file. Run the application, navigate to /cars, link on the Image link on one of the cars and post an image file. We haven’t included any user input checks but in a real application we would validate the user input. For now just make sure that you post an image file: jpg, jpeg, png, gif etc. We’ll show the file in the next post. Don’t worry if you don’t have images showing cars, that’s not the point.

If everything goes well then the Cars table should show the image IDs:

Image IDs shown in the Cars table

In the next post, which will be the last in this series on MongoDb, we’ll show and update the image.

View the posts related to data storage here.

MongoDB in .NET part 8: queries

Introduction

So far in this series we’ve gone through the most important database operations in MongoDb .NET: create, update and delete which was the topic of the previous post. We’ve also seen some query examples such as finding an element by ID.

In this post we’ll look more closely at querying in MongoDb .NET. The C# driver provides the following ways of querying documents:

  • The various Find methods of MongoCollection where you supply a Mongo query
  • LINQ: the current version of the C# driver supports LINQ queries
  • Aggregation: querying in a functional style

We’ll be working on the same demo web app as before in this series so have it open in Visual Studio.

The Find methods

We’ve already seen FindOneById when locating a single Car object:

Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));

We can use the same method to complete the Cars view. Run the web app and navigate to /cars. For each record in the cars list there are 3 links: Edit, Details, Delete. We’ve implemented the Edit and Delete functions but not the Details yet. Let’s do it.

Add the following action methid on CarsController:

public ActionResult Details(string id)
{
	Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
	return View(car.ConvertToViewModel());
}

Right-click “Details” and select Add View. Add the following View:

Add Details view for Car object

This will give you a simple table showing the details of the selected Car:

Car details show in Details view

The generic FindOneById method internally calls the FindOneByIdAs method which converts a document into an object. In case you need to convert a document into another object type then you can do it like this:

Customer customer = CarRentalContext.Cars.FindOneByIdAs<Customer>(new ObjectId("dfsdfs"));

This example is a bit stupid, I know, but you get the idea. You can use this method to construct different views of the same domain depending on the rights of the logged on user: FullCar, PartialCar, MinimalCar etc. The properties of each domain version will make sure that only the relevant fields are extracted from the document to construct the object:

LimitedCar limitedCar = CarRentalContext.Cars.FindOneByIdAs<LimitedCar>(new ObjectId("dfsdfs"));

There are other versions of the Find method that each return the first matching document:

  • FindOne(): finds the very first element in the collection
  • FindOne(IMongoQuery query): we’ve seen examples of how to construct an IMongoQuery. The Query object can be used to build query documents

Example: say you want to find the first Car whose rental costs less than or equal to 5:

IMongoQuery query = Query<Car>.LTE(c => c.DailyRentalFee, 5);
Car firstCheapCar = CarRentalContext.Cars.FindOne(query);

You can further refine your search in some overloads of the Find method where you can pass in a FindOneArgs object:

FindOneArgs findOneArgs = new FindOneArgs()
{
	Query = Query<Customer>.NE(c => c.Name, "samsung")
	, ReadPreference = new ReadPreference(ReadPreferenceMode.SecondaryPreferred)
};
Customer cust = CarRentalContext.Cars.FindOneAs<Customer>(findOneArgs);

Here we want to find the first customer whose name is NOT Samsung and we want to read from a secondary replica set member.

I encourage you to the explore the operators available on the Query object. Just type “Query.” or its generic counterpart in VisualStudio and IntelliSense will give you a long list of available options to build your query. There are so many that it’s not possible to cover them all in a single blog post. Keep in mind that you can always chain your queries with the And, Or and Not operators.

You can of course also locate several documents:

  • FindAll(): retrieves all documents in the collection
  • Find(IMongoQuery query): retrieves all documents matching the query

These two methods return a MongoCursor of type T. This object can be enumerated. Say you want to find all cars which have more than 3 doors:

IMongoQuery doorQuery = Query<Car>.GT(c => c.NumberOfDoors, 3);
MongoCursor<Car> bigCars = CarRentalContext.Cars.Find(doorQuery);

MongoCursor implements IEnumerable so you can perform the usual IEnumerable operations on it. E.g. the cursor can be enumerated:

foreach (Car c in bigCars)
{
     //do something
}

…or can also be turned into a List of cars:

List<Car> carsList = bigCars.ToList();

Like in the case of deferred operations in LINQ, the MongoDb Find query is not executed until you call an action on the cursor which enumerates it, such as iterating it or calling ToList.

Before the cursor is enumerated you can set extra options specific to MongoCursor, i.e. which are not part of IEnumerable. E.g. you can set the sort order as follows:

MongoCursor<Car> bigCars = CarRentalContext.Cars.Find(doorQuery);
IMongoSortBy sortByCars = SortBy<Car>.Descending(c => c.DailyRentalFee);
bigCars.SetSortOrder(sortByCars);
foreach (Car c in bigCars)
{
}

MongoCursor has a number of useful methods like that which all start with “Set”:

  • For pagination you can use the SetLimit(int limit) function in conjunction with SetSkip(int i). They behave like the Skip and Take LINQ operators
  • With SetFields you can provide the names of the fields to be returned in case you need to increase the performance. This has an effect similar to a limited SQL query: SELECT name, id FROM …

In case you want to know the number of matching documents without enumerating the cursor there’s the Count method:

MongoCursor<Car> bigCars = CarRentalContext.Cars.Find(doorQuery);
long howMany = bigCars.Count();

Let’s test the sorting method. In CarsController.Index we currently have the following contents:

public ActionResult Index()
{
	List<Car> carsInDb = CarRentalContext.Cars.FindAll().ToList();
        return View(carsInDb.ConvertAllToViewModels());
}

Change it to this so that we show the cars in a descending order by make:

public ActionResult Index()
{
	MongoCursor<Car> carsInDbCursor = CarRentalContext.Cars.FindAll();
        IMongoSortBy sortByCars = SortBy<Car>.Descending(c => c.Make);
	carsInDbCursor.SetSortOrder(sortByCars);
	return View(carsInDbCursor.ConvertAllToViewModels());
}

Run the app and you’ll see that the cars are indeed sorted according to the sort criteria. One thing to keep in mind is that the cursor is frozen as soon as the enumeration in begins by whatever enumeration operator. E.g. you cannot change the ordering of the cursor within a foreach loop.

LINQ

The MongoCollection object provides an AsQueryable extension method which is the starting point of building LINQ expressions:

MongoCursor<Car> carsInDbCursor = CarRentalContext.Cars.FindAll();
IEnumerable<Car> cars = carsInDbCursor.AsQueryable().Where(c => c.NumberOfDoors > 3);

Note that the Linq extensions in the MongoDb driver are evolving continuously. Not every query is possible to build by Linq that are found in the Query builder. In case you’re trying to run an unsupported Linq operator you’ll get an exception. You can read more about LINQ support in the driver here.

Aggregations

The aggregation mechanism in MongoDb provides not only search functions such as FindOne, FindAll etc. It also allows us to perform projections, groupings and summaries on documents to build a set of transformed documents.

You can combine transformation steps in a pipeline to build the final transformed document.

The entry point into creating aggregations is the Aggregate method of MongoCollection. The Aggregate method has a couple of overloads but the most flexible one accepts an AggregateArgs object. It returns an IEnumerable of BsonDocuments which is understandable. The transformed document is very unlikely to look like the original one so it cannot easily be deserialised into a domain object.

Taking the Car domain object we’ve been working with in this series we can have the following AggregateArgs to build a transformation pipeline:

AggregateArgs aggregateArgs = new AggregateArgs()
{
	Pipeline = new[]
	{
		new BsonDocument("$match", Query<Car>.LTE(c => c.DailyRentalFee, 10).ToBsonDocument())
		, new BsonDocument("$match", Query<Car>.GTE(c => c.DailyRentalFee, 3).ToBsonDocument())
		, new BsonDocument("$sort", new BsonDocument("DailyRentalFee", 1))
	}
};
IEnumerable<BsonDocument> documents = CarRentalContext.Cars.Aggregate(aggregateArgs);

This is a simple query but it’s a good starting point. We want to find the cars whose rental fee lies between 3 and 10 and then sort the documents by the rental fee in ascending order. For descending order use -1. This is also a good example to show that the transformation pipeline can be extended as much as you want in the BsonDocument array.

Aggregations can become quite involved. You’ll find lots of great examples on this website.

In the next post we’ll look at storing files in MongoDb.

View the posts related to data storage here.

MongoDB in .NET part 7: deleting documents

Introduction

In the previous post in this series we looked at how to update documents. So we now know how to insert, save and modify documents. We also need to be able to remove documents.

Remove and RemoveAll

Removing a document can be performed using the Remove method of IMongoCollection which has similar overloads to Update and returns a WriteConcernResult. However, while Update updates a single document by default even if there are multiple matching ones, Remove removes all matching documents. Most often we’ll remove a single document which matches an ID query but we can certainly construct an IMongoQuery which matches multiple documents. However, even if multiple documents are removed, the group of remove operations are not treated as a transaction. Each removal is a distinct operation.

You can supply a WriteConcern parameter which has the same purpose as in the case of Save and Update. You can also provide a RemoveFlags parameter which has 2 values: None and Single. With Single you can indicate that you only want to remove a single document if the query matches 2 or more documents. “None” simply means no flags which is the default value.

RemoveAll removes all documents in a collection while leaving indexes and metadata intact. There’s also a Drop method which is faster then RemoveAll but removes indexes and metadata too. If you need to remove the entire collection quickly then use the Drop method.

There’s also an atomic version called FindAndRemove which works in much the same way as FindAndUpdate we saw in the previous part.

Demo

We’ll extend the demo application we’ve been working on so far so have it ready in Visual Studio. This will be really simple actually. The Index.cshtml file of Cars already prepared a link for the Delete operation:

@Html.ActionLink("Delete", "Delete", new { id=item.Id })

We don’t yet have a Delete action so let’s add it to the CarsController:

public ActionResult Delete(string id)
{
	CarRentalContext.Cars.Remove(Query.EQ("_id", ObjectId.Parse(id)));
	return RedirectToAction("Index");
}

As you type Cars.Remove you’ll see the overloads of Remove where you can specify the parameters mentioned above. Run the application, navigate to /cars and press the Delete link on one of the items. The item should be removed from the list of items.

In the next part we’ll look more into MongoDb queries.

View the posts related to data storage here.

MongoDB in .NET part 6: updating documents

Introduction

In the previous part of this series we looked at write concerns, write acknowledgements and replacing documents with the Save method. We said that Save() works as an Insert if the document ID is not found. If the ID exists then the whole document is replaced with a new one.

The Update method

It is possible to update an existing document without replacing it using the Update method of MongoCollection. The Update method can be finetuned using its overloaded versions. The Save() method sends a completely new document to the Mongo server for replacement. The Update method instead sends an update document to the server containing one or more modification instructions. Update is used by the Save method behind the scenes. Our Save method in CarsController…

CarRentalContext.Cars.Save(modifiedCar);	

…can be mimicked by Update as follows:

CarRentalContext.Cars.Update(Query.EQ("_id", ObjectId.Parse(updateCarViewModel.Id)), Update.Replace(modifiedCar), UpdateFlags.Upsert);	

“Update” is a builder which helps build Update instructions – update documents – to the Mongo server. Notice the “Replace” update instruction which – as you may have guessed – will replace any existing document with a matching ID with the new one. Then we have the UpdateFlag where we specify an Upsert operation which we are familiar with by now. Skipping this flag would mean that a document is only updated if it exists by the given ID.

The Update builder contains a lot of operators. The operators return an IMongoUpdate object which represents a modification document and can be used in the Update method as a parameter. Just type “Update.” in Visual Studio and IntelliSense will show you about 20 different ones. Examples:

  • Inc: increments a numeric field – int, double, long – by a specified value
  • Rename: change the name of a field, e.g. from “Make” to “Type” in our Car domain object
  • Set: change the value of a field or insert it if it doesn’t exist, e.g. change 12 to 15 of the field DailyRentalFee
  • Unset: remove a field
  • Push: an array operation which allows us to append a value to an array
  • PopFirst and PopLast: removes the first or the last element of an array
  • Pull: remove specific elements from an array
  • AddToSet: add item to an array if it doesn’t exist

Some array operators have versions with “Each” in the method name. They allow to pass in several values e.g. in the AddToSetEach method. These operators are all fluent ones, i.e. you can chain them together to create a composite IMongoUpdate document.

“Update” is not the only way to build update documents. The following statements are all equivalent:

Update.Set("price", 1);
new UpdateBuilder().Set("price", 1);
Update<Car>.Set(c => c.DailyRentalFee, 2);
new UpdateBuilder<Car>().Set(c => c.DailyRentalFee, 3);

You will probably want to use the strongly typed versions to avoid hard coded string values.

So if you want to specifically update the rental fee of a Car object you can write as follows:

CarRentalContext.Cars.Update(Query.EQ("_id", ObjectId.Parse(updateCarViewModel.Id)), Update<Car>.Set(c => c.DailyRentalFee, 12));

By default the Update method will only update the first document that matches the query. If you want all documents to be updated that match the query then the Multi flag must be given. To change the fee of all Ford cars you can write as follows:

CarRentalContext.Cars.Update(Query.EQ("Make", "Ford"), Update<Car>.Set(c => c.DailyRentalFee, 2), UpdateFlags.Multi);

How to update then?

We’ve now seen two ways of updating a document: Save and Update. Here are some considerations:

  • Save is much more compact: you type less, it’s easier to test, is more robust than the Update equivalent
  • Update results in a more procedural style of code whereas Save is more OOP
  • Replacing a document has more overhead than modifying it, so Update performs better. The Save method fetches a document, modifies it and puts it in place of the original
  • While the Save method performs these steps we might run into concurrency issues. To exclude the possibility of data corruption during an update you may want to go for atomic updates, e.g. the FindAndModify method
  • Update allows for some very fine-grained modifications. They can be very useful in a data migration scenario, especially the Multi updates

We mentioned the FindAndModify atomic method above. It returns a FindAndModifyResult and accepts a FindAndModifyArgs object. Example:

FindAndModifyArgs args = new FindAndModifyArgs()
{
	Query = Query.EQ("_id", ObjectId.Parse(updateCarViewModel.Id))
	,Update = Update<Car>.Set(c => c.DailyRentalFee, 3)
	,Upsert = false
	,SortBy = SortBy<Car>.Ascending(c => c.Id)
	,VersionReturned = FindAndModifyDocumentVersion.Original
};
FindAndModifyResult res = CarRentalContext.Cars.FindAndModify(args);

You’ll recognise the Query, Upsert and Update properties. FindAndModify always modifies a single document. If there are more matching documents then the SortBy property can be used to determine which document will be updated: first or last. In the above example we’re querying on the ID field so this shouldn’t be an issue. The FindAndModifyResult return object includes a ModifiedDocument property of type BsonDocument. It represents the modified document but the exact representation depends on the VersionReturned argument. If it’s set to “Original” then ModifiedDocument will show the document before the modification. If it’s set to “Modified” then the modified document will be returned. The point is that if you run the Update method then and then query for the same document then someone else might have modified the same document. With FindAndModify you can avoid this as it can return the document in the way that you have modified it. The BsonDocument can be deserialised with the GetModifiedDocumentAs method of the FindAndModifyResult object.

Miscellaneous

  • A document is limited to 16MB – this can hold a very large domain. However, always consider how large your domain can grow especially with nested arrays
  • Atomic updates to documents are atomic per document. If you batch update several documents then each document will be updated atomically one by one but the whole operation is not atomic, i.e. we have no transactions
  • There’s no built-in mechanism to check for concurrent access to a document. The EntityFramework object context ensures that each operation is carried out in a thread-safe manner. There’s no equivalent of this context object in MongoDb so it’s possible that while you’re updating a document someone else deletes it. Or you change the rental fee to 2 and someone else updates it to 3 at the same time. The update to be processed last will win. However, this is nothing new – we could easily fill a whole textbook with concurrency issues in databases

In the next post we’ll look at how to delete documents.

View the posts related to data storage here.

MongoDB in .NET part 5: WriteConcerns and replacing documents

Introduction

In the previous part of this series we successfully inserted a new Car object in our database. We also managed to read all items from the database and show them in a table. We’ll continue looking at MongoDb operations.

We’ll keep working on the same demo as before so have it ready in Visual Studio.

WriteConcern

Recall that we inserted a new Car object via the MongoDb Insert method of the MongoCollection object. It returns a WriteConcernResult object that we haven’t bothered with. In the Create action method of the CarsController controller modify…

CarRentalContext.Cars.Insert(car);

…to…

WriteConcernResult writeResult = CarRentalContext.Cars.Insert(car);
bool ok = writeResult.Ok;

Insert a breakpoint within the action method, start the app, navigate to /cars/create and insert a new Car object. When the code execution stops inspect the “ok” boolean value in Visual Studio. It should be true, meaning that the write operation has succeeded. We could read this result because by default the driver will wait for an acknowledgement if some CRUD operation is carried out through the MongoClient object. There’s one more level to enhance durability – you can wait for a journal commit acknowledgement from the database. You can achieve this by either extending the connection string to…

mongodb://localhost/?journal=true

…or in code as follows:

String mongoHost = ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString;
MongoClientSettings settings =
	MongoClientSettings.FromUrl(new MongoUrl(mongoHost));	
settings.WriteConcern.Journal = true;
_mongoClient = new MongoClient(settings);

The WriteConcern property has at least one more important property you should know about: W. That’s right, a property called W. It is of type WValue and is represented by an integer. Say you have a cluster of MongoDb databases – a replica set – with one primary server for writes and 1 or more secondary servers for reads. The write operation will be propagated to all secondary servers in a matter of milliseconds. This is what you often see in real life database environments to enhance durability and data availability – if one server dies then you still have at least one more for reads and writes. The property W indicates the number of nodes in the cluster that must acknowledge the write – or update – operation. A similar mechanism from SQL Server is AlwaysOn. WriteConcern currently has 4 predefined values:

settings.WriteConcern = WriteConcern.W1;
settings.WriteConcern = WriteConcern.W2;
settings.WriteConcern = WriteConcern.W3;
settings.WriteConcern = WriteConcern.W4;

W1 means at least 1 node has to acknowledge the modification. You understand the rest. There is a special property where the majority of the nodes must acknowledge the operation:

settings.WriteConcern = WriteConcern.WMajority;

There’s another special value if you want to skip write acknowledgements altogether:

settings.WriteConcern = WriteConcern.Unacknowledged;

Update the CarRentalContext constructor to the following:

String mongoHost = ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString;
MongoClientSettings settings =
MongoClientSettings.FromUrl(new MongoUrl(mongoHost));
settings.WriteConcern = WriteConcern.Unacknowledged;
_mongoClient = new MongoClient(settings);			
_mongoServer = _mongoClient.GetServer();			
_mongoDatabase = _mongoServer.GetDatabase(ConfigurationManager.AppSettings["CarRentalDatabaseName"]);

Run the app and try to enter a new car. Press Create and then you should get a null pointer exception here:

bool ok = writeResult.Ok;

The writeResult object will be null as we didn’t want any acknowledgement. This is called fire-and-forget.

Why would you want to relax the acknowledgment mechanism? I can give you an example from my work. In one of our projects we monitor some real-time performance statistics during load testing of a web site and show it in various graphs. The process is very “insert” intensive with lots of inserts during the test. We don’t really care if a couple of data points are lost here and there as there are always new ones coming in. So to save time and get better performance we relax all types of write acknowledgment. Also, the stats are only interesting during the live testing so we’re not much concerned with durability either. You can even set the acknowledgement policy on the level of the operation, e.g.:

CarRentalContext.Cars.Insert(car, WriteConcern.Unacknowledged);

…or on the collection level:

public MongoCollection<Car> Cars
{
	get
	{
		return CarRentalDatabase.GetCollection<Car>("cars", WriteConcern.W1);
	}
}

However, for most types of data in a database you’ll want to turn on write acknowledgments. Change the CarRentalContext to the default write concern:

public CarRentalContext()
{
	String mongoHost = ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString;
	MongoClientSettings settings =
		MongoClientSettings.FromUrl(new MongoUrl(mongoHost));			
	_mongoClient = new MongoClient(settings);			
	_mongoServer = _mongoClient.GetServer();			
	_mongoDatabase = _mongoServer.GetDatabase(ConfigurationManager.AppSettings["CarRentalDatabaseName"]);
}

Updates

Updates can be performed in 2 ways:

  • Replace an existing document with a new one
  • Modify an existing document without removing it

The Save() method will replace an existing document. Save() can also act as an Insert method: if there’s no document with the specified ID or no ID is provided at all then Save will create a new document. This is known as an upsert, i.e. update or insert. Save() returns a WriteConcernResult like Insert. There’s also another method called Update which Save calls upon internally. However, Save is more concise and saves you some typing.

The View for /cars created a table for us to show the cars. It has also prepared some links to update a record – check the Edit link. Don’t click on it yet as have no action method or controller for it.

In CarsController add the following Edit stub:

[HttpGet]
public ActionResult Edit(string id)
{

}

We’ll need to find the car with the selected ID and retrieve it from the database. To find an object by id you can use the FindOneById method on the collection like this:

[HttpGet]
public ActionResult Edit(string id)
{
      Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
}

You can build very sophisticated search criteria using the IMongoQuery builder. We can write the above query as follows:

IMongoQuery findByIdQuery = Query.EQ("_id", new ObjectId(id));
Car car = CarRentalContext.Cars.FindOne(findByIdQuery);

Query is used to build IMongoQuery objects. Just type “Query.” in VS and you’ll see lots of familiar properties such as “EQ” for equal, GT – greater than, LT – less than – and lots more. Whenever you need build a query to the Find or FindOne methods you’ll use the Query object. In case you need to specify a range of criteria you can combine them with And or Or:

Query.And(Query.EQ("someProperty", propertyValue), Query.EQ("otherProperty", otherValue));

Let’s continue. We’ll need a View for the Edit action and a corresponding view-model. Add a class called UpdateCarViewModel to the ViewModels folder:

public class UpdateCarViewModel
{
	[Editable(false)]
	public string Id { get; set; }
	[Required]
	[Display(Name = "Type of car")]
	public string Make { get; set; }
	[Required]
	[Display(Name = "Number of doors")]
	[Range(2, 6)]
	public int NumberOfDoors { get; set; }
	[Required]
	[Display(Name = "Daily rental fee")]
	public decimal DailyRentalFee { get; set; }
	[Required]
	[Display(Name = "List of allowed countries delimited with ';'")]
	public string DelimitedListOfCountries { get; set; }
}

It’s almost the same as the InsertCarViewModel, we could probably do with some subclassing, but it’s fine as it is for the demo. We’ll need the ID so that the Save method can locate the correct document. Next we’ll need to convert a Car object into an UpdateCarViewModel object. Insert the following extension method to DomainExtensions.cs:

public static UpdateCarViewModel ConvertToUpdateViewModel(this Car carDomain)
{
	UpdateCarViewModel updateVm = new UpdateCarViewModel()
	{
		Id = carDomain.Id
		, DailyRentalFee = carDomain.DailyRentalFee
		, Make = carDomain.Make
		, NumberOfDoors = carDomain.NumberOfDoors
	};

	if (carDomain.CountriesAllowedIn != null && carDomain.CountriesAllowedIn.Count() > 0)
	{
		StringBuilder sb = new StringBuilder();
		for (int i = 0; i < carDomain.CountriesAllowedIn.Count(); i++)
		{
			sb.Append(carDomain.CountriesAllowedIn.ElementAt(i));
			if (i < carDomain.CountriesAllowedIn.Count() - 1)
			{
				sb.Append(";");
			}
		}
		updateVm.DelimitedListOfCountries = sb.ToString();
	}

	return updateVm;
}

Back in CarsController modify the Edit GET method as follows:

[HttpGet]
public ActionResult Edit(string id)
{
	Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
	return View(car.ConvertToUpdateViewModel());
}

Right-click “Edit”, select Add View:

Adding Edit template to Car object

Again, VS will create a basic Edit template for the selected item. Let’s check if it works so far. Run the application, navigate to /cars and click on the Edit link for one of the Car objects. This should open the following Edit form:

Edit Car template populated

Now we need to write the POST Edit action to complete the loop. Add the following stub to the CarsController:

[HttpPost]
public ActionResult Edit(UpdateCarViewModel updateCarViewModel)
{

}

We’ll need another extension method to convert an UpdateCarViewModel back to a Car. Add the following to DomainExtensions:

public static Car ConvertToDomain(this UpdateCarViewModel updateCarViewModel)
{
	Car car = new Car()
	{
		Id = updateCarViewModel.Id
		, DailyRentalFee = updateCarViewModel.DailyRentalFee
		, Make = updateCarViewModel.Make
		, NumberOfDoors = updateCarViewModel.NumberOfDoors
	};
	string[] countries = updateCarViewModel.DelimitedListOfCountries.Split(new char[] { ';' }, StringSplitOptions.RemoveEmptyEntries);
	car.CountriesAllowedIn = countries.ToList();
	return car;
}

Again, we’ll need the Id so that the Save function can find the document to be modified. Here’s the final version of the POST Edit method:

[HttpPost]
public ActionResult Edit(UpdateCarViewModel updateCarViewModel)
{
	if (ModelState.IsValid)
	{
		Car modifiedCar = updateCarViewModel.ConvertToDomain();
		CarRentalContext.Cars.Save(modifiedCar);
		return RedirectToAction("Index");
	}
	return View(updateCarViewModel);
}

We didn’t have to look up the Car object in the database – the Save method will check the ID for us. Run the application, update an existing Car and save it. If all works fine then you’ll be redirected to the Index page with the updated value in the list of Cars.

In the next post we’ll look at updates a bit closer.

View the posts related to data storage here.

MongoDB in .NET part 4: the POCO model, insertions, indexes and capped collections

Introduction

We saw the basics of POCO documents in the previous post of this series. In this part we’ll build up our model with POCOs.

Open the demo application CarRentalWeb we started out with and let’s get to work!

Our model

Add a new folder to the solution called Domain. In that folder insert a class called Car:

public class Car
{
        [BsonRepresentation(MongoDB.Bson.BsonType.ObjectId)]
	public string Id { get; set; }
	public string Make { get; set; }
	[BsonRepresentation(MongoDB.Bson.BsonType.Double)]
	public decimal DailyRentalFee { get; set; }
	public int NumberOfDoors { get; set; }
	public List<string> CountriesAllowedIn { get; set; }
}

You’ll recognise the BsonRepresentationAttribute from the previous post. You may be wondering why we have public string Id with the BsonRepresentation attribute. We saw in the previous post that Mongo stores the id field – called “_id” – in its own ObjectId format. We could simply write…

public ObjectId Id { get; set; }

…but ObjectId might cause problems with data binding elsewhere in the application as it is not a native .NET object. So we choose its nearest .NET approximation which is a string.

While we’re on this topic the above object definition may not even be considered strictly POCO anymore. It contains attributes that belong to a technology that a true domain object should not care about. These attributes have to do with data persistence and domain objects should have no knowledge of how they are represented in the data store. If you’re not sure what I mean I recommend that you go through the series on Domain Driven Design to learn how to separate domain and persistence models to keep the domain models strictly POCO. At the same time you may be fine with decorating your domain objects like that but it’s your decision.

However, this series is about MongoDb, not DDD, so we’ll allow for such deviations. You can then organise your objects the way you want in your project.

Inserting cars

As we said before a Mongo collection is a group of related documents. It is similar to a table in a relational database. Mongo collections are represented by the MongoCollection object in the C# driver and can be accessed via the MongoDatabase object. In our CarRentalContext context class we built such an object in the class constructor.

Add a new folder to the solution called ViewModels. ViewModels here are not the same as the behaviour-rich objects in MVVM. Instead they are simple DTOs – data transfer objects – to represent the “View” part of our models in MVC. Essentially they will be bound to the elements of the form where we insert new cars. Add the following class to the ViewModels folder:

public class InsertCarViewModel
{
	[Required]
	[Display(Name = "Type of car")]
	public string Make { get; set; }
	[Required]
	[Display(Name = "Number of doors")]
	[Range(2,6)]
	public int NumberOfDoors { get; set; }
	[Required]
	[Display(Name = "Daily rental fee")]
	public decimal DailyRentalFee { get; set; }
	[Required]
	[Display(Name = "List of allowed countries delimited with ';'")]
	public string DelimitedListOfCountries { get; set; }
}

This prepares some basic validation and display values using standard MVC annotations. Let’s prepare the Cars controller. Add a new empty MVC controller like this…:

Add cars controller in Visual Studio

Insert a Create action method like this:

[HttpGet]
public ActionResult Create()
{
	return View();
}

Right-click “Create” and select “Add View”. Fill in the form like this:

Create GET Create template for Car view model

In case the Model class list doesn’t show the InsertCarViewModel then close the window and build the project. Click OK and you’ll be presented an out-of-the box Create view. Start the application and navigate to /cars/create. You should see a simple form:

GET Create form to insert new cars

The VS 2013 equivalent will have a different style but that’s not important. When we press the Edit button the POST Create action will be called. Let’s insert it into the CarsController:

[HttpPost]
public ActionResult Create(InsertCarViewModel insertCarViewModel)
{
	if (ModelState.IsValid)
	{
	}
	return View(insertCarViewModel);
}

It’s not doing anything yet. We want to test if the incoming InsertCarViewModel object is correctly populated. Insert a breakpoint at the if statement. Start the application, navigate to /cars/create, fill in some valid values and press create. Code execution will stop at the breakpoint. Inspect the insertCarViewModel in VS – you’ll see that all properties have been populated correctly. We’ll stay on the same page after the Create method returns. The next step is to actually insert this record into MongoDb.

We’ll need to convert the viewmodel to its proper domain representation. We’ll use an extension method to do that. Insert a class called DomainExtensions in the Domain folder. Change the namespace to CarRentalWeb. Declare the class as static and insert the following static method in it:

namespace CarRentalWeb
{
	public static class DomainExtensions
	{
		public static Car ConvertToDomain(this InsertCarViewModel insertCarViewModel)
		{
			Car car = new Car()
			{
				DailyRentalFee = insertCarViewModel.DailyRentalFee
				, Make = insertCarViewModel.Make
				, NumberOfDoors = insertCarViewModel.NumberOfDoors
			};
			string[] countries = insertCarViewModel.DelimitedListOfCountries.Split(new char[] { ';' }, StringSplitOptions.RemoveEmptyEntries);
			car.CountriesAllowedIn = countries.ToList();
			return car;
		}
	}
}

Back in Create(InsertCarViewModel insertCarViewModel) call this extension method as follows:

if (ModelState.IsValid)
{
	Car car = insertCarViewModel.ConvertToDomain();
}

We’ll need access to the database context in this controller – and most likely in all our controllers. Add a new class called BaseController to the Controllers folder. Select the “class” template, not the New Controller options.

public class BaseController : Controller
{
	private CarRentalContext _carRentalContext;

	public BaseController()
	{
		_carRentalContext = new CarRentalContext();
	}

	public CarRentalContext CarRentalContext
	{
		get
		{
			return _carRentalContext;
		}
	}
}

Change the declaration of CarsController to inherit from the BaseController:

public class CarsController : BaseController

Next we’ll extend the CarRentalContext class to return a mongo collection that holds the Cars objects. Open CarRentalContext.cs and add the following getter:

public MongoCollection<Car> Cars
{
	get
	{
		return CarRentalDatabase.GetCollection<Car>("cars");
	}
}

The MongoCollection class has an overloaded version where you can specify the type of objects stored in the collection. The MongoDatabase object is used to get hold of a collection of a specific type. We need to specify the name of the collection – “cars” – which is an arbitrary string that you can specify, it could as well be “MickeyMouse”, but you probably prefer something descriptive. The type specification will help the serialisation mechanism to convert to and from Car and its BSON representation.

Back in CarsController.Create we’ll use the simplest approach to insert a new document. Add the following line after the conversion from the view model:

CarRentalContext.Cars.Insert(car);
return RedirectToAction("Index");

That’s as simple as it gets. We’ll come back to the Insert method in the next post as it returns a WriteConcern object which deserves more attention. We can ignore it for the time being.

After an insertion we’d like to return to the Index page which lists all the Car objects in a table, which is why we have the RedirectToAction call. The CarsController already has an Index method which returns a View(). Let’s tie the View to an IEnumerable of Car view-model objects. We don’t have those yet so add a new class called CarViewModel to the ViewModels folder:

public class CarViewModel
{	
        [Display(Name = "ID")]
	public string Id { get; set; }
	public string Make { get; set; }
	[Display(Name = "Rental fee per day")]
	public decimal DailyRentalFee { get; set; }
	[Display(Name = "Number of doors")]
	public int NumberOfDoors { get; set; }
	[Display(Name = "Allowed countries")]
	public string CountriesAllowedIn { get; set; }
}

We’ll need to convert the Car objects from the database into the CarViewModel representations. Add the following extension methods to DomainExtensions.cs.:

public static IEnumerable<CarViewModel> ConvertAllToViewModels(this IEnumerable<Car> carDomains)
{
	foreach (Car car in carDomains)
	{
		yield return car.ConvertToViewModel();
	}
}

public static CarViewModel ConvertToViewModel(this Car carDomain)
{
	CarViewModel carViewModel = new CarViewModel()
	{
		Id = carDomain.Id
		, DailyRentalFee = carDomain.DailyRentalFee
		, Make = carDomain.Make
		, NumberOfDoors = carDomain.NumberOfDoors
	};

	if (carDomain.CountriesAllowedIn != null && carDomain.CountriesAllowedIn.Count() > 0)
	{
		StringBuilder sb = new StringBuilder();
		for (int i = 0; i < carDomain.CountriesAllowedIn.Count(); i++)
		{
			sb.Append(carDomain.CountriesAllowedIn.ElementAt(i));
			if (i < carDomain.CountriesAllowedIn.Count() - 1)
			{
				sb.Append(",");
			}
		}
		carViewModel.CountriesAllowedIn = sb.ToString();
	}
        return carViewModel;
}

The CarsController Index action will need to fetch all the car objects, convert them to view models and show them to the viewer. Compile the solution and right-click “Index”. Select Add View…:

Add view to list all car objects

This will give you a basic template to show all items in the sequence in a table.

We can use the FindAll method of MongoCollection to retrieve all items without any search parameters:

public ActionResult Index()
{
	List<Car> carsInDb = CarRentalContext.Cars.FindAll().ToList();
        return View(carsInDb.ConvertAllToViewModels());
}

We should be good to go. Start the application, navigate to /cars/create. Fill in the form with valid data, e.g.:

Create car form values example

If everything’s OK then you’ll see the new Car object listed in its CarViewModel representation on the Index page:

Car created and index page shown

That wasn’t too difficult, right? We didn’t even need to create a database or collection. They were both created for us upon the first insert.

Indexes

You’ll probably know that it’s important to set an index on fields that often figure in searches. Here’s how you can set a composite index on the Car object on the Make and NumberOfDoors properties – which probably doesn’t make sense, this is only example code:

IndexKeysBuilder<Car> carIndexBuilder = IndexKeys<Car>.Ascending(c => c.Make, c => c.NumberOfDoors);
IndexOptionsBuilder<Car> carIndexOptions = IndexOptions<Car>.SetName("Car_CompositeIndex").SetTimeToLive(new TimeSpan(2, 0, 0, 0));
CarRentalContext.Cars.EnsureIndex(carIndexBuilder, carIndexOptions);

With SetTimeToLive you can set an expiry date on a collection.

You can drop an Index by its name as follows:

CarRentalContext.Cars.DropIndexByName("Car_CompositeIndex");

Capped collections

A capped collection is a collection which is not allowed to grow beyond a specific size in bytes. If the maximum size is reached then the elements will be dropped starting from the first element in a FIFO fashion. Here’s how you can set the cap to 50MB:

CollectionOptionsBuilder optionsBuilder = new CollectionOptionsBuilder();
optionsBuilder.SetCapped(true);
optionsBuilder.SetMaxSize(52428800);
CarRentalContext.CarRentalDatabase.CreateCollection("NewCollection", optionsBuilder);

Read the next part of the series here.

View the posts related to data storage here.

MongoDB in .NET part 3: starting with POCO documents

Introduction

In the previous post we set up our MongoDb server and saw how to establish a connection to it. We’ll now start inserting objects into the database. In other words we’ll create MongoDb documents.

Note that there are basically two ways to create documents: either via low-level BSON objects or your “proper” custom POCO ones that you create in your domain layer. In this tutorial we’ll not cover BSON documents. I think most developers prefer the more strongly typed POCO way of working with documents. With BSON you have to declare your object structure through strings.

Here’s a code example of creating a BSON document in C# from the C# driver tutorial:

BsonDocument document = new BsonDocument 
{
    { "name", name },
    { "city", city }, // not added if city is null
    { "dob", dob, dobAvailable } // not added if dobAvailable is false
};

Another example:

BsonDocument nested = new BsonDocument {
{ "name", "John Doe" },
    { "address", new BsonDocument {
        { "street", "123 Main St." },
        { "city", "Centerville" },
        { "state", "PA" },
        { "zip", 12345}
    }}
};

We’ll only look at POCOs here. I think you’ll agree that it’s easier to work with “real” objects for all CRUD operations and queries.

Data serialisation

The C# driver will translate to and from BSON and POCO automatically through data (de)serialisation. The default serialisation is very simple. Say we have the following POCO:

public class Customer
{
	public string Name { get; set; }
	public string Address { get; set; }
}

The default JSON representation of this POCO will be:

{
    "Name" : "Elvis"
    , "Address" : "Graceland"
}

The POCO will be saved in the BSON format but the above JSON helps visualise the binary source. If the POCO has some sequence of objects, e.g.:

public class Customer
{
	public string Name { get; set; }
	public string Address { get; set; }
        IEnumerable<string> Telephones { get; set; }
}

The it’s translated into a JSON array:

{
    "Name" : "Elvis"
    , "Address" : "Neverland"
    , "Telephones" : ["123", "456"]
}

Nested objects such as…

public class Customer
{
	public string Name { get; set; }
	public string Address { get; set; }
	IEnumerable<string> Telephones { get; set; }
	public WebPage PublicPage { get; set; }
}

public class WebPage
{
	public bool IsSsl { get; set; }
	public string Domain { get; set; }
}

…are represented as nested documents like here:

{
    "Name" : "Elvis"
    , "Address" : "Neverland"
    , "Telephones" : ["123", "456"]
    , "PublicPage" : { "IsSsl": true, "Domain" : "company.com" }
}

I think you’re beginning to see the modelling advantage of MongoDb compared to RMDBS databases. You can have any kind of object structure, the resulting document will immediately store it in the appropriate JSON/BSON format without any extra mapping, schema changes, data migrations etc. Feel free to describe your business model in POCOs with no constraints laid by the database. Have nested objects, lists, arrays etc. inside your domain objects – MongoDb documents will be able to handle them with no extra effort.

Changing the default serialisation

Occasionally you may want to deviate from the standard serialisation mechanism: you can ignore certain properties or define a different property name. Serialisation attributes in MongoDb will help you a great deal here. They are similar to the DataMember, IgnoreMember etc. attributes in .NET serialisation. Examples:

To ignore a property use BsonIgnore:

public class Customer
{
	public string Name { get; set; }
	public string Address { get; set; }
	IEnumerable<string> Telephones { get; set; }
	[BsonIgnore]
	public WebPage PublicPage { get; set; }
}

The PublicPage property will not be serialised in the BSON document.

To store a property under a different name use the BsonElement attribute:

public class Customer
{		
	public string Name { get; set; }
	public string Address { get; set; }
	IEnumerable<string> Telephones { get; set; }
	[BsonElement("PublicWebPage")]
	public WebPage PublicPage { get; set; }
}

You might want to do this if the Customer object had a property called PublicWebPage before but was renamed to PublicPage. You still want all the old records to be deserialised so you can declare the name of the property like that.

Ignore NULL values:

public class Customer
{		
	public string Name { get; set; }
	[BsonIgnoreIfNull]
	public string Address { get; set; }
}

By default private fields are not serialised. You can use the empty BsonElement to include it in the serialisation:

public class Customer
{		
	public string Name { get; set; }
	public string Address { get; set; }
	IEnumerable<string> Telephones { get; set; }
	[BsonElement]
	private WebPage PublicPage { get; set; }
}

Declare how the property will be represented in BSON, i.e. under which type, using the BsonRepresentation attribute:

public class Customer
{
	[BsonRepresentation(MongoDB.Bson.BsonType.Double)]
	public decimal TotalOrders { get; set; }
}

The .NET decimal type has no equivalent in BSON, it’s treated as a string which makes sorting and comparing inefficient. It’s better to cast it to a BSON double which is close enough.

We saw that the unique ID field is called “_id” by default and is of type ObjectId. You can declare the ID field using the BsonId attribute:

public class Customer
{
	[BsonId]
	public int CustomerId { get; set; }
}

The ID will still be serialised as “_id” in the document but you can refer to it as “CustomerId” in the POCO. Keep in mind, that it’s your responsibility to create unique integer IDs – there’s no equivalent of SQL Server’s auto-increment (1,1) in MongoDb. In case you stick to the ObjectId type then MongoDb will create unique values for you.

Dates are always a different beast with all the time zones and formats. MongoDb translates all local dates to UTC automatically when a date field is serialised. However, it can create problems when the date is deserialised. You’ll by default get the UTC date whereas you may expect the local one. I think it’s always wise to work only with UTC in your domains and convert dates for views only, but you can use the BsonDateTimeOptions attribute to override the default behaviour:

public class Customer
{
	[BsonDateTimeOptions(Kind = DateTimeKind.Local)]
	public DateTime CustomerSince { get; set; }
}

This way the UTC date stored in the document will be converted to the local time zone of the computer.

In case you only want to store the “date” part of a date, i.e. without the hours:minutes:seconds bit then use the DateOnly parameter of BsonDateTimeOptions:

public class Customer
{
	[BsonDateTimeOptions(Kind = DateTimeKind.Local, DateOnly = true)]		
	public DateTime CustomerSince { get; set; }
}

If you remove a property from your POCO then the document will adjust automatically during serialisation as we said before. However, the C# driver won’t like it when deserialising the BSON representation. It will throw a FileFormatException in case a property in the BSON document doesn’t match any of the properties in the POCO. You can make the driver ignore unmapped properties using the BsonIgnoreExtraElements class-level attribute:

[BsonIgnoreExtraElements]
public class Customer
{
	[BsonDateTimeOptions(Kind = DateTimeKind.Local, DateOnly = true)]		
	public DateTime CustomerSince { get; set; }
}

In the next post we’ll start building the POCOs for our demo car rental application.

View the posts related to data storage here.

MongoDB in .NET part 2: setup

Introduction

In the previous post we covered the basics of MongoDb. In this post we’ll look at how to set it up in Windows and how to connect to it through the C# driver.

Note that the C# driver is actually a .NET driver which can be called from Visual Basic as well.

Setup on Windows

Let’s set up MongoDb on the localhost. Navigate to the MongoDb downloads page and click the Windows download link:

MongoDb Windows install package

This will download an msi installer. Step through the installer. Choose the “typical” installation mode when the question comes. The installer will put the MongoDb files in the Program Files/MongoDB x.x Standard folder:

MongDb default installation folder

The installation package is self-contained, i.e. there’s no dependency on other elements. We’ll put it somewhere where it’s easier to find. Create a folder called mongodb on your C drive and cut/paste the contents of the Program Files/MongoDB x.x Standard into it. MongoDb will by default use a folder called “data” on your C drive to store the documents. So insert a folder called “data” on c:\. Within the “data” folder create another folder called “db”.

Let’s first connect to MongoDb manually. Open a command prompt, navigate to c:\mongodb\bin and issue the following command:

mongod.exe

This will start the MongoDb database engine. You’ll see that it’s waiting for connections:

Start MondoDb database

Open up another command prompt, navigate to c:\mongodb\bin and start the client:

mongo.exe

Start MongoDb client in console window

So we managed to connect to the MongoDb engine from a client. You can write direct JavaScript commands in the client to create databases, collections, insert documents etc. I will not go through these low level operations here – I’m only familiar with the basics myself anyway. We’ll do all that in C#. However, if you’re interested in those direct commands then you can check out the official manual.

Close both command prompts to start afresh and open a new one. Make sure it has administrator rights. It’s not convenient having to start the DB engine all the time we want to talk to it. Fortunately MongoDb can be set up as a Windows service. We’ll need a folder for the log files so add a folder called “log” to c:\mongodb. Create a configuration file for the logpath option for MongoDb in the command prompt by issuing this command:

echo logpath=C:\mongodb\log\mongo.log> C:\mongodb\mongod.cfg

This will create a mongod.cfg file in c:\mongodb.

Next run the following command:

C:\mongodb\bin\mongod.exe –config C:\mongodb\mongod.cfg –install

This will create the MongoDB Windows service. Note that it is a double-dash in front of “config” and “install”, this is not easily visible due to the rendering of this post on your screen. Let’s start the service:

net start mongodb

The command prompt should say that the service was started successfully. If you get an error message that the service doesn’t exist then re-run the installation command from an Administrator command prompt.

Let’s connect to MongoDb again from the client:

c:\mongodb\bin\mongo.exe

It should say “connecting to: test” so we’ve connected to the MongoDb engine that is now up and running thanks to the Windows service. “Test” is the default database that the mongo client connects to first.

Connecting to MongoDb in code

We’ll build up a simple MVC web application where people can rent cars.

Warning: I’ll be concentrating on Mongo-related functionality and ignore software practices like SOLID or layered architecture. If you are familiar with those concepts then you’ll see a lot of “big no-no” code here. However, I want to concentrate on MongoDb. It’s your responsibility to organise your code to fit good software engineering design.

Open Visual Studio 2012/2013 and create a new ASP.NET MVC web application called CarRentalWeb. It doesn’t matter if it’s MVC4 or 5 really, we’ll be concentrating on MongoDb. I’ll build this demo using MVC 4 in VS 2012 but MVC 5 in VS 2013 should be equally fine. If you have VS 2013 then when you create the project make sure to select the “No authentication” auth type. We don’t want to be sidetracked by security issues. Add the following NuGet package to the references list:

MongoDb C# driver from NuGet

A general note: the MongoDb C# driver evolves quite quickly with lots of additions and deprecated code. At the time of writing this post the most recent version was 1.9.1. By the time you’re reading this there may well be a new version. It’s happened before to me that an updated release included code that immediately broke the existing code I wrote. It is possible that the code I’m going to write in this demo gets outdated quickly but hopefully it’ll be enough for you to follow along.

The NuGet package will install the following libraries:

MongoDb libraries added by NuGet

Just like an SQL Server driver such as EntityFramework the MongoDb driver will also need a connection string. There’s a whole list of examples on how to construct MongoDb connection strings here. Open web.config and add the following connection string to the connectionStrings section:

<add name="CarRentalConnectionString" connectionString="mongodb://localhost"/>

This is probably as easy as it gets.

Open HomeController.cs in the Controllers folder and add the following private fields and a constructor to it:

private MongoClient _mongoClient;
private MongoServer _mongoServer;
private MongoDatabase _mongoDatabase;

public HomeController()
{
	_mongoClient = new MongoClient(ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString);
	_mongoServer = _mongoClient.GetServer();
	_mongoDatabase = _mongoServer.GetDatabase(ConfigurationManager.AppSettings["CarRentalDatabaseName"]);
}

The MongoServer object will be used to connect to a database. MongoClient serves for sending commands to the server. We’ll store the name of the database in web.config. Add the following setting to the appSettings section:

<add key="CarRentalDatabaseName" value="CarRentalDatabase"/>

We haven’t created that database anywhere but we don’t need to worry. It will be created for us automatically when we insert the first document. Let’s verify that the connection can be made through reading the server build info. Erase the contents of Index() and add the following instead:

_mongoDatabase.GetCollectionNames();
return Json(_mongoDatabase.Server.BuildInfo, JsonRequestBehavior.AllowGet);

The first row is only needed to trigger a connection in the driver, it doesn’t have any other purpose. OK, let’s now run the application. If everything went well then you’ll get some long JSON on your screen:

MongoDb build info in JSON format

Note the MongoDb server version: 2.6.1. Also, the C# driver we installed above was of version 1.9. All demos and code examples will be built on this configuration. If you have an older version of whichever then you might not get the same behaviour, e.g. you may see that an older version of the driver doesn’t have certain methods. Therefore make sure you have those versions are higher.

A short aside: MongoDb settings

We saw above the simplest way to declare our connection to the MongoDb server. There’s however a rich imperative way of declaring your MongoDb settings through the MongoClientSettings object. Examples:

MongoClientSettings settings = new MongoClientSettings();
String mongoHost = ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString);
MongoServerAddress address = new MongoServerAddress(mongoHost);
settings.ConnectTimeout = new TimeSpan(0, 1, 5);
settings.SocketTimeout = new TimeSpan(0, 1, 5);
settings.Server = address;

The above code sets the connection string and the connection timeouts. You can also set the connection pool and wait queue size with a wait queue timeout:

settings.MaxConnectionPoolSize = 1500;
settings.WaitQueueSize = 1500;
settings.WaitQueueTimeout = new TimeSpan(0, 1, 0);

It’s very likely that our database will require authentication. The following method can help you set up the MongoDb credentials list:

private static List<MongoCredential> GetMongoCredentials()
{
	string username = ConfigurationManager.AppSettings["MongoDbUserName"];
	string password = ConfigurationManager.AppSettings["MongoDbPassword"];
	string mongoDbAuthMechanism = ConfigurationManager.AppSettings["MongoDbAuthMechanism"];
	MongoInternalIdentity internalIdentity = new MongoInternalIdentity(LtpMongoDbDefaultDatabase, username);
	PasswordEvidence passwordEvidence = new PasswordEvidence(password);
	MongoCredential mongoCredential = new MongoCredential(mongoDbAuthMechanism, internalIdentity, passwordEvidence);
	List<MongoCredential> credentials = new List<MongoCredential>() { mongoCredential };
	return credentials;
}

…where “MongoDbAuthMechanism” has the following value: “MONGODB-CR”, more about that here. You can then set the credentials as follows:

settings.Credentials = GetMongoCredentials();

In case you have a replica set in place, i.e. a primary server with one ore more secondary servers then you can set the read preference and connection mode as follows:

settings.ReadPreference = new ReadPreference(ReadPreferenceMode.SecondaryPreferred);
settings.ConnectionMode = ConnectionMode.ReplicaSet;

And finally we construct the MongoServer object as follows:

MongoClient client = new MongoClient(settings);
MongoServer mongoServer = client.GetServer();
_mongoServer = mongoServer;	

The MongoClientSettings object has more properties than shown here. If you need to set some specific property to connect to your database then check out this object, it will probably have something for you.

The MongoDb context class

If you’ve used Linq to SQL or EntityFramework then you’ll know that they each have their DB context classes. There’s no equivalent in MongoDb but it can be emulated to a certain extent. We’ll wrap the code which constructs the MongoDatabase class in its own container class. Insert a new folder called MongoDb to the project. Add a class called CarRentalContext to it:

private MongoClient _mongoClient;
private MongoServer _mongoServer;
private MongoDatabase _mongoDatabase;

public CarRentalContext()
{
	_mongoClient = new MongoClient(ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString);
	_mongoServer = _mongoClient.GetServer();
	_mongoDatabase = _mongoServer.GetDatabase(ConfigurationManager.AppSettings["CarRentalDatabaseName"]);
}

public MongoDatabase CarRentalDatabase
{
	get
	{
		return _mongoDatabase;
	}
}

This way the context class can be accessed from other controllers as well. Keep in mind that this context class is not nearly as sophisticated as the DB context classes of Linq to SQL or EntityFramework. There’s no SubmitChanges/SaveChanges method to carry out the changes stored in the context. There’s no InsertOnSubmit, DeleteOnSubmit etc. methods either. The changes will be immediate unless you write your own Unit of Work wrapper around this simple context class. Back in HomeController erase the private fields and delete the contents of the constructor. Update the code as follows:

private CarRentalContext _carRentalContext;

public HomeController()
{
	_carRentalContext = new CarRentalContext();
}

public ActionResult Index()
{
	_carRentalContext.CarRentalDatabase.GetCollectionNames();
	return Json(_carRentalContext.CarRentalDatabase.Server.BuildInfo, JsonRequestBehavior.AllowGet);
}

I can’t stress this enough: you should never build real-life controllers like that. Make sure you understand SOLID which tells you how to inject such dependencies correctly. Re-run the application to make sure it’s still working. It should.

You’ll notice that there’s no code to connect to and disconnect from the database. The driver will handle all that for you through the connection pool. Also, MongoServer instances are cached for a given set of settings so don’t worry about using this context class multiple times. The driver will provide the cached version. Also, the 3 key objects we saw above, i.e. MongoClient, MongoServer and MongoDatabase are all threadsafe, so don’t worry about locks and data races.

In the next post we’ll be looking into (de)serialisation of POCO objects in MongoDb.

You can view all posts related to data storage on this blog here.

MongoDB in .NET part 1: foundations

Introduction

There has been a strong increase in the usage of document based NoSql databases in the previous years. One prominent example is MongoDb. The default choice for storing data in a .NET project has most often been SQL Server. While SQL Server is probably still the most popular choice .NET developers can choose from other well-tested alternatives depending on their project needs. There is a whole suite of NoSQL databases out there besides MongoDb. Some examples:

At the time of writing this post MongoDb was the most popular NoSQL database according to db-engines. Even if you’re a staunch relational DBMS advocate and think you’ll never use anything else for your data storage solution it’s still beneficial to know that there are other tools out there. Also, you can have have a mixed data storage strategy where you store some objects in a relational DBMS and some others in a document store depending on the objects’ nature and the advantages and disadvantages of each storage mechanism.

You know that “we’ve always done it this way” is a dangerous mindset. Studying MongoDb can widen your skills set and offer a different point of view to solving data storage challenges.

In this series we’ll explore MongoDb in more details and how it can used in a .NET project.

MongoDb terms

As hinted at above MongoDb is a document-based database. In SQL Server the records are stored in rows where each column usually represents a property of the object that the table describes. In MongoDb however, information is stored in JSON documents – or to be exact, in Binary JSON, i.e. BSON. Here’s an example from the MongoDb homepage:

{
"_id": 1,
"name": {
"first": "John",
"last": "Backus"
},
"contribs": ["Fortran",
"ALGOL",
"Backus-Naur Form",
"FP"],
"awards": [{
"award": "W.W. McDowell Award",
"year": 1967,
"by": "IEEE Computer Society"
},
{
"award": "Draper Prize",
"year": 1993,
"by": "National Academy of Engineering"
}]
}

You’ll immediately notice that the properties such as “awards” can be arrays where each element in turn has their own properties. If you’re familiar with JSON then this is no surprise to you. As the documents are stored in binary format you cannot just open a BSON document in a text editor however.

The above JSON/BSON string is an example of a document. It consists of a set of key-value pairs such as _id = 1. Each key-value pair is an element or a field. You can see that “name” itself is a document within the larger document, also called an embedded document. A group of related documents is called a collection. A collection is similar to a table in SQL Server.

Each document has a unique ID field denoted by “_id”. It has the same function as a unique primary key in an SQL Server table. It can be of any type really but it’s customary to make it of type ObjectId which looks similar to a GUID. No matter which type you use, the ID must be unique across the collection of documents.

Advantages

There are some key advantages of MongoDb compared to traditional “table-based” databases:

  • Dynamic data structure with flexible schemas: you don’t need to define columns and tables. You can in fact store pretty much anything within the same collection
  • Due to the lack of strict schemas data migrations become a lot easier too: if you change your domain structure, i.e. you business objects in code, the document will store the objects correspondingly. You can force a change in the schema through changing your custom objects automatically
  • You don’t need separate tables to show relationships: the above JSON structure shows you that there’s no need for a separate “awards” collection, you can show the relationship directly within the document – there’s no need for foreign keys and constraints. If you extract a single item from a collection then you’ll immediately get its associated objects: order with all order items, rock band with all concerts, making it a breeze to perform operations on those linked objects
  • MongoDb documents therefore allow storing your objects in an object oriented fashion which is sometimes difficult and awkward to solve in SQL Server with separate tables and keys
  • Due to the lack of constraints such as secondary keys updating and deleting items will be easier: there’s no cascade delete, no orphans
  • Speed: MongoDb is very fast and efficient in querying and inserting items in a collection

Disadvantages

All of the above is very well and good but there are couple of things that you need to be aware of if you’re coming from an SQL Server environment – which probably at least 95% of .NET developers do.

  • Lack of professional tools: with SQL Server you can use SSMS for some very advanced GUI-based database operations, such as database profiling, SQL jobs, a query editor, IntelliSense and a whole lot more. There’s no equivalent in MongoDb. If you want to work directly with collections without one of the drivers – C#, Java, Php etc – then you’ll need to write your code in a console window or develop a custom solution yourself. There’s one visual tool though that you can use to improve the administration of your MongoDb database: RoboMongo
  • As of writing this post MongoDb doesn’t support transactions
  • Many developers will say that the lack of a schema is actually a disadvantage: you cannot associate objects through keys, you cannot force a compulsory data structure like “NOT NULL”
  • No stored procedures
  • Difficult to retrieve lost data

You can read more about the advantages and disadvantages of MongoDb here.

Drivers

While you can interact with MongoDb directly in a command window you’ll most certainly prefer to do that in code through a client library, a.k.a a driver. There are drivers available for most mainstream languages, like C#, Java or PHP. You can check out the full list of language-specific drivers here.

In the next post we’ll look at how to set up MongoDb and how to connect to it from code.

You can view all posts related to data storage on this blog here.

ultimatemindsettoday

A great WordPress.com site

Elliot Balynn's Blog

A directory of wonderful thoughts

HarsH ReaLiTy

A Good Blog is Hard to Find

Softwarearchitektur in der Praxis

Wissenswertes zu Webentwicklung, Domain-Driven Design und Microservices

Technology Talks

on Microsoft technologies, Web, Android and others

Software Engineering

Web development

Disparate Opinions

Various tidbits

chsakell's Blog

WEB APPLICATION DEVELOPMENT TUTORIALS WITH OPEN-SOURCE PROJECTS

Once Upon a Camayoc

Bite-size insight on Cyber Security for the not too technical.

Guru N Guns's

OneSolution To dOTnET.

Johnny Zraiby

Measuring programming progress by lines of code is like measuring aircraft building progress by weight.

%d bloggers like this: