MongoDB in .NET part 10: other file operations

Introduction

In the previous post on MongoDb we talked about inserting files to GridFS and linking them to a Car object. In this post we’ll look at how to read, delete and update a file.

We’ll build on the CarRental demo project we’ve been working on, so have it ready in Visual Studio.

Reading a file

The primary way of reading a file with the C# driver of MongoDb is the Download method of MongoGridFS and its numerous overloads. With Download you can extract the contents of a file to a stream or to the local file system. With the MongoGridFS.Open method you can put the contents of a file to a Stream but you can only locate the file by file name. There are also the MongoGridFS.Find methods – Find, FindOne, FindAll, FindOneById which allows you to find metadata on a file and indirectly open the contents of the file through the MongoGridFSFileInfo object that the Find methods return. MongoGridFSFileInfo has methods to open a file: Open, OpenRead and OpenText. The Find method lets you search for a specific file using an IMongoQuery object. These Find methods are very much the same as what we saw in the post on querying documents.

Deleting and updating a file

Deletes and updates are handled under the same section as there’s no separate update method. An update means first removing a file and then inserting a new one instead.

Demo

Let’s show the image associated with the Car object on Image.cshtml if one exists. We’ll need a helper method on the CarViewModel object to determine whether it has an image. Add the following property to CarViewModel.cs:

public bool HasImage
{
	get
	{
		return !string.IsNullOrEmpty(ImageId);
	}
}

We’ll retrieve the Image from a controller action. Add a new Controller to the Controllers folder called ImagesController. Select the Empty MVC Controller template type. Make it derive from BaseController and insert an Image action method:

public class ImagesController : BaseController
{        
        public ActionResult Image(string imageId)
        {
		MongoGridFSFileInfo imageFileInfo = CarRentalContext.CarRentalDatabase.GridFS.FindOneById(new ObjectId(imageId));
		return File(imageFileInfo.OpenRead(), imageFileInfo.ContentType);
        }
}

The last missing piece is to extend Image.cshtml to show the image. Add the following markup just below the closing brace of the Html.BeginForm statement:

@if (Model.HasImage)
{
	<img src="@Url.Action("Image", "Images", new { imageId = @Model.ImageId})" />
}

Run the application, navigate to /cars and click on the Image link of a Car which has a valid image file. If everything’s gone fine then you should see the associated image:

Show car image from GridFS

In case you’re wondering: that’s right, I uploaded the Windows logo from the background of my computer as the car image. It doesn’t make any difference what image you’ve associated with the Car, the main thing is that it’s shown correctly.

Deleting and updating a file

As hinted above, there’s no separate update function for files. Therefore files are updated in two steps: delete the existing one and insert a new one instead. We’ve already seen how to insert a file and link it to an object, so we only need to consider deletions in this section.

Deletions are performed with the Delete function and its overloads: delete by a query, delete by file name and delete by id. You can also delete a file by its MongoGridFSFileInfo wrapper, an instance of which we’ve seen above.

If you use the Delete(IMongoQuery) overload then all files will be deleted one by one that match the search criteria, i.e. the matching files are not deleted in an atomic operation.

We’ll extend our demo app as follows: when an image is uploaded then we check if the Car already has one. If so, then the image is replaced. Otherwise we’ll upload the file like we saw in the previous post.

Insert the following method to CarsController:

private void DeleteCarImage(Car car)
{
	CarRentalContext.CarRentalDatabase.GridFS.DeleteById(car.ImageId);
	car.ImageId = string.Empty;
	CarRentalContext.Cars.Save(car);
}

We first delete the image by its ID. Then we need to set the ImageID property of the car to an empty string as there’s no automatic mechanism that can determine that the image of a Car has been deleted and update the ImageId property. Lastly we save the Car object.

We can call this function in an extended version of POST Image:

[HttpPost]
public ActionResult Image(string id, HttpPostedFileBase file)
{
	Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
	if (!string.IsNullOrEmpty(car.ImageId))
	{
		DeleteCarImage(car);
	}
	AttachImageToCar(file, car);
	return RedirectToAction("Index");
}

That’s it. Run the app as usual and try to replace an image attached to a Car, it should go fine.

Other interesting operations

The MongoGridFS and MongoGridFSFileInfo objects have a MoveTo function. They don’t actually move anything, they set the file name metadata field to a different value. If you want to duplicate a file, then call the CopyTo method that’s available on each object.

You can also modify the metadata of a file through the SetMetada method of MongoGridFS.

The end

This was the last post in the series on MongoDb in .NET. There’s of course loads more to look at but we’d need a book to cover all aspects. However, this should be enough for you to start coding and explore MongoDb in further detail on your own.

View the posts related to data storage here.

MongoDB in .NET part 9: storing files

Introduction

In the previous post we looked at a couple of query techniques in the MongoDb C# driver. Before that in this series we saw how to store, update, delete and query “usual” objects like Car. Storing files, such as a PDF or Word file, on the other hand is a different matter. They take up a lot of space in the data store and require special data types to store the bytes.

In this post we’ll look at how files can be stored in MongoDb using its GridFS technology. GridFS stores files in MongoDb documents just like normal objects. However, it stores the file contents in chunks of 256KB. Each chunk is stored in a MongoDb document. Recall that a single MongoDocument can store 16MB of data. Below 16MB you can still opt for storing the byte array contents of a file in a single MongoDb document but for consistency you probably should store all your files in GridFS. GridFS also stores the file metadata in a separate MongoDocument with appropriate links to the constituent chunks.

GridFS is similar to a file system such as the Windows file directory. However, it is independent of the platform it’s running on, i.e. GridFS is not constrained by any limitations of the file system of the OS.

Also, as each chunk is stored in a MongoDb document, GridFS prepares the way for content replication in a Replica Set scenario.

Atomic operations are not available for files in GridFS. You cannot locate and update a file in a single step. Also, the only way to update a file is by replacing an existing one. You cannot send an Update document to the Mongo server to update a small part of a file. So GridFS is probably a good option for files that don’t change too often.

GridFS in the C# driver

GridFS is represented by the GridFS property of MongoDatabase. The GridFS instance with the default settings can be reached in our demo CarRental demo as follows:

MongoGridFS gridFsDefault = CarRentalContext.CarRentalDatabase.GridFS;

You can use the GetGridFS method of MongoDatabase to override the default settings:

MongoGridFSSettings gridFsSettings = new MongoGridFSSettings();
gridFsSettings.ChunkSize = 1024;
MongoGridFS gridFsCustom = CarRentalContext.CarRentalDatabase.GetGridFS(gridFsSettings);

The files can be retrieved using the Files property of MongoGridFS. By default it returns a collection of BsonDocuments but it can be serialised into MongoGridFSFileInfo objects. They allow us to easily extract the metadata about a file in an object oriented way: file name, size in bytes, date uploaded, content type etc. As the files stored in GridFS are independent of the local file system many of these properties can be provided by the user, such as the file name or a list of aliases.

If you ever wish to view individual chunks of a file then you can retrieve them using the Chunks property:

MongoCollection<BsonDocument> chunks = gridFsDefault.Chunks;

These cannot be serialised into any strongly types object yet. Each BsonDocument has a files_id field by which you can identify the chunks belonging to a single file. You can get the file id from the MongoGridFSFileInfo object. You can put the chunks into the right order using the numeric ‘n’ field which denotes the place of a chunk in the sequence. The binary data can be extracted using the ‘data’ field.

We can upload files using one of the overloaded Upload functions of MongoGridFS. One such overload allows us to specify a Stream, a remote file name – to change the file name representation in the metadata – and a MongoGridFSCreateOptions object. This last parameter will let us further define the options for uploading a single file: a list of aliases, chunk size, content type, custom metadata, upload date and an id. The Upload function returns a MongoGridFSFileInfo object.

Another way to upload files is to use the Create and CreateText methods of MongoDatabase.

Demo

We’ll extend our CarRental demo. The first goal is to be able to upload an image linked to a Car. Locate Index.cshtml in the Views/Cars folder. It contains a set of links for each Car entry: Edit, Details and Delete. Add the Image action link to that list:

<td>
        @Html.ActionLink("Edit", "Edit", new { id=item.Id }) |
        @Html.ActionLink("Details", "Details", new { id=item.Id }) |
        @Html.ActionLink("Delete", "Delete", new { id=item.Id }) |
       	@Html.ActionLink("Image", "Image", new { id=item.Id })
</td>

Add the following method to CarsController.cs:

public ActionResult Image(string id)
{
	Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
	return View(car.ConvertToViewModel());
}

Right-click “Image” and select Add View…:

Add car rental image view

You’ll get an empty cshtml file. We’ll fill it out in a little bit. We’ll certainly need a POST action in CarsController that accepts the posted file and the Car id. Also, we’ll need a property on the Car object to store the image id. OK, let’s start from the bottom. Add the following property to Car:

public string ImageId { get; set; }

Add it to CarViewModel.cs as well:

[Display(Name = "Image ID")]
public string ImageId { get; set; }

We won’t yet see the image itself until the next post so we’ll only show the image id first. Locate ConvertToViewModel in DomainExtensions.cs and add the new property to the conversion:

CarViewModel carViewModel = new CarViewModel()
{
	Id = carDomain.Id
	, DailyRentalFee = carDomain.DailyRentalFee
	, Make = carDomain.Make
	, NumberOfDoors = carDomain.NumberOfDoors
	, ImageId = carDomain.ImageId
};

In Views/Cars/Index.cshtml we’ll extend the table to view the image IDs:

...
<th>
	@Html.DisplayNameFor(model => model.ImageId)
</th>
...

<td>
	@Html.DisplayFor(modelItem => item.ImageId)
</td>

...

Run the application to test if it’s still working. The image IDs will be empty of course in the Cars table. Click the Image link to see if it leads us to the empty Image view.

In CarsController add the following private method:

private void AttachImageToCar(HttpPostedFileBase file, Car car)
{
	ObjectId imageId = ObjectId.GenerateNewId();
	car.ImageId = imageId.ToString();
	CarRentalContext.Cars.Save(car);
	MongoGridFSCreateOptions createOptions = new MongoGridFSCreateOptions()
	{
		Id = imageId
		, ContentType = file.ContentType
	};
	CarRentalContext.CarRentalDatabase.GridFS.Upload(file.InputStream, file.FileName, createOptions);
}

We construct a new object id and set it as the ImageId property of the Car object. We then call the Save function to update the Car in the database. Then we use an overload of the Upload function to upload the file to GridFS. Note how we got hold of the default GridFS instance through the GridFS property. As we specify the ID and the content type ourselves we can use the MongoGridFSCreateOptions to convey the values. Call this function from the following POST method in CarsController:

[HttpPost]
public ActionResult Image(string id, HttpPostedFileBase file)
{
	Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
	AttachImageToCar(file, car);
	return RedirectToAction("Index");
}

We can now fill in Image.cshtml:

@model CarRentalWeb.ViewModels.CarViewModel

@{
    ViewBag.Title = "Image";
}

<h2>Car image</h2>

@using (Html.BeginForm(null, null, FormMethod.Post, new { enctype = "multipart/form-data" }))
{
	@Html.AntiForgeryToken()

	<div>
        
		<div>
			<label>Make: </label>
			<div>
				@Model.Make
			</div>
		</div>
		
		<div>
			<label>Image: </label>
			<div>
				<input type="file" id="file" name="file"/>
			</div>
		</div>

		<div>
			<div>
				<input type="submit" value="Save"/>
			</div>
		</div>
	</div>
}

<div>
	@Html.ActionLink("Back to Cars", "Index")
</div>

Specifying multipart form data in “enctype = multipart/form-data” will enable us to post the entire form data along with the posted file. Run the application, navigate to /cars, link on the Image link on one of the cars and post an image file. We haven’t included any user input checks but in a real application we would validate the user input. For now just make sure that you post an image file: jpg, jpeg, png, gif etc. We’ll show the file in the next post. Don’t worry if you don’t have images showing cars, that’s not the point.

If everything goes well then the Cars table should show the image IDs:

Image IDs shown in the Cars table

In the next post, which will be the last in this series on MongoDb, we’ll show and update the image.

View the posts related to data storage here.

MongoDB in .NET part 8: queries

Introduction

So far in this series we’ve gone through the most important database operations in MongoDb .NET: create, update and delete which was the topic of the previous post. We’ve also seen some query examples such as finding an element by ID.

In this post we’ll look more closely at querying in MongoDb .NET. The C# driver provides the following ways of querying documents:

  • The various Find methods of MongoCollection where you supply a Mongo query
  • LINQ: the current version of the C# driver supports LINQ queries
  • Aggregation: querying in a functional style

We’ll be working on the same demo web app as before in this series so have it open in Visual Studio.

The Find methods

We’ve already seen FindOneById when locating a single Car object:

Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));

We can use the same method to complete the Cars view. Run the web app and navigate to /cars. For each record in the cars list there are 3 links: Edit, Details, Delete. We’ve implemented the Edit and Delete functions but not the Details yet. Let’s do it.

Add the following action methid on CarsController:

public ActionResult Details(string id)
{
	Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
	return View(car.ConvertToViewModel());
}

Right-click “Details” and select Add View. Add the following View:

Add Details view for Car object

This will give you a simple table showing the details of the selected Car:

Car details show in Details view

The generic FindOneById method internally calls the FindOneByIdAs method which converts a document into an object. In case you need to convert a document into another object type then you can do it like this:

Customer customer = CarRentalContext.Cars.FindOneByIdAs<Customer>(new ObjectId("dfsdfs"));

This example is a bit stupid, I know, but you get the idea. You can use this method to construct different views of the same domain depending on the rights of the logged on user: FullCar, PartialCar, MinimalCar etc. The properties of each domain version will make sure that only the relevant fields are extracted from the document to construct the object:

LimitedCar limitedCar = CarRentalContext.Cars.FindOneByIdAs<LimitedCar>(new ObjectId("dfsdfs"));

There are other versions of the Find method that each return the first matching document:

  • FindOne(): finds the very first element in the collection
  • FindOne(IMongoQuery query): we’ve seen examples of how to construct an IMongoQuery. The Query object can be used to build query documents

Example: say you want to find the first Car whose rental costs less than or equal to 5:

IMongoQuery query = Query<Car>.LTE(c => c.DailyRentalFee, 5);
Car firstCheapCar = CarRentalContext.Cars.FindOne(query);

You can further refine your search in some overloads of the Find method where you can pass in a FindOneArgs object:

FindOneArgs findOneArgs = new FindOneArgs()
{
	Query = Query<Customer>.NE(c => c.Name, "samsung")
	, ReadPreference = new ReadPreference(ReadPreferenceMode.SecondaryPreferred)
};
Customer cust = CarRentalContext.Cars.FindOneAs<Customer>(findOneArgs);

Here we want to find the first customer whose name is NOT Samsung and we want to read from a secondary replica set member.

I encourage you to the explore the operators available on the Query object. Just type “Query.” or its generic counterpart in VisualStudio and IntelliSense will give you a long list of available options to build your query. There are so many that it’s not possible to cover them all in a single blog post. Keep in mind that you can always chain your queries with the And, Or and Not operators.

You can of course also locate several documents:

  • FindAll(): retrieves all documents in the collection
  • Find(IMongoQuery query): retrieves all documents matching the query

These two methods return a MongoCursor of type T. This object can be enumerated. Say you want to find all cars which have more than 3 doors:

IMongoQuery doorQuery = Query<Car>.GT(c => c.NumberOfDoors, 3);
MongoCursor<Car> bigCars = CarRentalContext.Cars.Find(doorQuery);

MongoCursor implements IEnumerable so you can perform the usual IEnumerable operations on it. E.g. the cursor can be enumerated:

foreach (Car c in bigCars)
{
     //do something
}

…or can also be turned into a List of cars:

List<Car> carsList = bigCars.ToList();

Like in the case of deferred operations in LINQ, the MongoDb Find query is not executed until you call an action on the cursor which enumerates it, such as iterating it or calling ToList.

Before the cursor is enumerated you can set extra options specific to MongoCursor, i.e. which are not part of IEnumerable. E.g. you can set the sort order as follows:

MongoCursor<Car> bigCars = CarRentalContext.Cars.Find(doorQuery);
IMongoSortBy sortByCars = SortBy<Car>.Descending(c => c.DailyRentalFee);
bigCars.SetSortOrder(sortByCars);
foreach (Car c in bigCars)
{
}

MongoCursor has a number of useful methods like that which all start with “Set”:

  • For pagination you can use the SetLimit(int limit) function in conjunction with SetSkip(int i). They behave like the Skip and Take LINQ operators
  • With SetFields you can provide the names of the fields to be returned in case you need to increase the performance. This has an effect similar to a limited SQL query: SELECT name, id FROM …

In case you want to know the number of matching documents without enumerating the cursor there’s the Count method:

MongoCursor<Car> bigCars = CarRentalContext.Cars.Find(doorQuery);
long howMany = bigCars.Count();

Let’s test the sorting method. In CarsController.Index we currently have the following contents:

public ActionResult Index()
{
	List<Car> carsInDb = CarRentalContext.Cars.FindAll().ToList();
        return View(carsInDb.ConvertAllToViewModels());
}

Change it to this so that we show the cars in a descending order by make:

public ActionResult Index()
{
	MongoCursor<Car> carsInDbCursor = CarRentalContext.Cars.FindAll();
        IMongoSortBy sortByCars = SortBy<Car>.Descending(c => c.Make);
	carsInDbCursor.SetSortOrder(sortByCars);
	return View(carsInDbCursor.ConvertAllToViewModels());
}

Run the app and you’ll see that the cars are indeed sorted according to the sort criteria. One thing to keep in mind is that the cursor is frozen as soon as the enumeration in begins by whatever enumeration operator. E.g. you cannot change the ordering of the cursor within a foreach loop.

LINQ

The MongoCollection object provides an AsQueryable extension method which is the starting point of building LINQ expressions:

MongoCursor<Car> carsInDbCursor = CarRentalContext.Cars.FindAll();
IEnumerable<Car> cars = carsInDbCursor.AsQueryable().Where(c => c.NumberOfDoors > 3);

Note that the Linq extensions in the MongoDb driver are evolving continuously. Not every query is possible to build by Linq that are found in the Query builder. In case you’re trying to run an unsupported Linq operator you’ll get an exception. You can read more about LINQ support in the driver here.

Aggregations

The aggregation mechanism in MongoDb provides not only search functions such as FindOne, FindAll etc. It also allows us to perform projections, groupings and summaries on documents to build a set of transformed documents.

You can combine transformation steps in a pipeline to build the final transformed document.

The entry point into creating aggregations is the Aggregate method of MongoCollection. The Aggregate method has a couple of overloads but the most flexible one accepts an AggregateArgs object. It returns an IEnumerable of BsonDocuments which is understandable. The transformed document is very unlikely to look like the original one so it cannot easily be deserialised into a domain object.

Taking the Car domain object we’ve been working with in this series we can have the following AggregateArgs to build a transformation pipeline:

AggregateArgs aggregateArgs = new AggregateArgs()
{
	Pipeline = new[]
	{
		new BsonDocument("$match", Query<Car>.LTE(c => c.DailyRentalFee, 10).ToBsonDocument())
		, new BsonDocument("$match", Query<Car>.GTE(c => c.DailyRentalFee, 3).ToBsonDocument())
		, new BsonDocument("$sort", new BsonDocument("DailyRentalFee", 1))
	}
};
IEnumerable<BsonDocument> documents = CarRentalContext.Cars.Aggregate(aggregateArgs);

This is a simple query but it’s a good starting point. We want to find the cars whose rental fee lies between 3 and 10 and then sort the documents by the rental fee in ascending order. For descending order use -1. This is also a good example to show that the transformation pipeline can be extended as much as you want in the BsonDocument array.

Aggregations can become quite involved. You’ll find lots of great examples on this website.

In the next post we’ll look at storing files in MongoDb.

View the posts related to data storage here.

MongoDB in .NET part 7: deleting documents

Introduction

In the previous post in this series we looked at how to update documents. So we now know how to insert, save and modify documents. We also need to be able to remove documents.

Remove and RemoveAll

Removing a document can be performed using the Remove method of IMongoCollection which has similar overloads to Update and returns a WriteConcernResult. However, while Update updates a single document by default even if there are multiple matching ones, Remove removes all matching documents. Most often we’ll remove a single document which matches an ID query but we can certainly construct an IMongoQuery which matches multiple documents. However, even if multiple documents are removed, the group of remove operations are not treated as a transaction. Each removal is a distinct operation.

You can supply a WriteConcern parameter which has the same purpose as in the case of Save and Update. You can also provide a RemoveFlags parameter which has 2 values: None and Single. With Single you can indicate that you only want to remove a single document if the query matches 2 or more documents. “None” simply means no flags which is the default value.

RemoveAll removes all documents in a collection while leaving indexes and metadata intact. There’s also a Drop method which is faster then RemoveAll but removes indexes and metadata too. If you need to remove the entire collection quickly then use the Drop method.

There’s also an atomic version called FindAndRemove which works in much the same way as FindAndUpdate we saw in the previous part.

Demo

We’ll extend the demo application we’ve been working on so far so have it ready in Visual Studio. This will be really simple actually. The Index.cshtml file of Cars already prepared a link for the Delete operation:

@Html.ActionLink("Delete", "Delete", new { id=item.Id })

We don’t yet have a Delete action so let’s add it to the CarsController:

public ActionResult Delete(string id)
{
	CarRentalContext.Cars.Remove(Query.EQ("_id", ObjectId.Parse(id)));
	return RedirectToAction("Index");
}

As you type Cars.Remove you’ll see the overloads of Remove where you can specify the parameters mentioned above. Run the application, navigate to /cars and press the Delete link on one of the items. The item should be removed from the list of items.

In the next part we’ll look more into MongoDb queries.

View the posts related to data storage here.

MongoDB in .NET part 6: updating documents

Introduction

In the previous part of this series we looked at write concerns, write acknowledgements and replacing documents with the Save method. We said that Save() works as an Insert if the document ID is not found. If the ID exists then the whole document is replaced with a new one.

The Update method

It is possible to update an existing document without replacing it using the Update method of MongoCollection. The Update method can be finetuned using its overloaded versions. The Save() method sends a completely new document to the Mongo server for replacement. The Update method instead sends an update document to the server containing one or more modification instructions. Update is used by the Save method behind the scenes. Our Save method in CarsController…

CarRentalContext.Cars.Save(modifiedCar);	

…can be mimicked by Update as follows:

CarRentalContext.Cars.Update(Query.EQ("_id", ObjectId.Parse(updateCarViewModel.Id)), Update.Replace(modifiedCar), UpdateFlags.Upsert);	

“Update” is a builder which helps build Update instructions – update documents – to the Mongo server. Notice the “Replace” update instruction which – as you may have guessed – will replace any existing document with a matching ID with the new one. Then we have the UpdateFlag where we specify an Upsert operation which we are familiar with by now. Skipping this flag would mean that a document is only updated if it exists by the given ID.

The Update builder contains a lot of operators. The operators return an IMongoUpdate object which represents a modification document and can be used in the Update method as a parameter. Just type “Update.” in Visual Studio and IntelliSense will show you about 20 different ones. Examples:

  • Inc: increments a numeric field – int, double, long – by a specified value
  • Rename: change the name of a field, e.g. from “Make” to “Type” in our Car domain object
  • Set: change the value of a field or insert it if it doesn’t exist, e.g. change 12 to 15 of the field DailyRentalFee
  • Unset: remove a field
  • Push: an array operation which allows us to append a value to an array
  • PopFirst and PopLast: removes the first or the last element of an array
  • Pull: remove specific elements from an array
  • AddToSet: add item to an array if it doesn’t exist

Some array operators have versions with “Each” in the method name. They allow to pass in several values e.g. in the AddToSetEach method. These operators are all fluent ones, i.e. you can chain them together to create a composite IMongoUpdate document.

“Update” is not the only way to build update documents. The following statements are all equivalent:

Update.Set("price", 1);
new UpdateBuilder().Set("price", 1);
Update<Car>.Set(c => c.DailyRentalFee, 2);
new UpdateBuilder<Car>().Set(c => c.DailyRentalFee, 3);

You will probably want to use the strongly typed versions to avoid hard coded string values.

So if you want to specifically update the rental fee of a Car object you can write as follows:

CarRentalContext.Cars.Update(Query.EQ("_id", ObjectId.Parse(updateCarViewModel.Id)), Update<Car>.Set(c => c.DailyRentalFee, 12));

By default the Update method will only update the first document that matches the query. If you want all documents to be updated that match the query then the Multi flag must be given. To change the fee of all Ford cars you can write as follows:

CarRentalContext.Cars.Update(Query.EQ("Make", "Ford"), Update<Car>.Set(c => c.DailyRentalFee, 2), UpdateFlags.Multi);

How to update then?

We’ve now seen two ways of updating a document: Save and Update. Here are some considerations:

  • Save is much more compact: you type less, it’s easier to test, is more robust than the Update equivalent
  • Update results in a more procedural style of code whereas Save is more OOP
  • Replacing a document has more overhead than modifying it, so Update performs better. The Save method fetches a document, modifies it and puts it in place of the original
  • While the Save method performs these steps we might run into concurrency issues. To exclude the possibility of data corruption during an update you may want to go for atomic updates, e.g. the FindAndModify method
  • Update allows for some very fine-grained modifications. They can be very useful in a data migration scenario, especially the Multi updates

We mentioned the FindAndModify atomic method above. It returns a FindAndModifyResult and accepts a FindAndModifyArgs object. Example:

FindAndModifyArgs args = new FindAndModifyArgs()
{
	Query = Query.EQ("_id", ObjectId.Parse(updateCarViewModel.Id))
	,Update = Update<Car>.Set(c => c.DailyRentalFee, 3)
	,Upsert = false
	,SortBy = SortBy<Car>.Ascending(c => c.Id)
	,VersionReturned = FindAndModifyDocumentVersion.Original
};
FindAndModifyResult res = CarRentalContext.Cars.FindAndModify(args);

You’ll recognise the Query, Upsert and Update properties. FindAndModify always modifies a single document. If there are more matching documents then the SortBy property can be used to determine which document will be updated: first or last. In the above example we’re querying on the ID field so this shouldn’t be an issue. The FindAndModifyResult return object includes a ModifiedDocument property of type BsonDocument. It represents the modified document but the exact representation depends on the VersionReturned argument. If it’s set to “Original” then ModifiedDocument will show the document before the modification. If it’s set to “Modified” then the modified document will be returned. The point is that if you run the Update method then and then query for the same document then someone else might have modified the same document. With FindAndModify you can avoid this as it can return the document in the way that you have modified it. The BsonDocument can be deserialised with the GetModifiedDocumentAs method of the FindAndModifyResult object.

Miscellaneous

  • A document is limited to 16MB – this can hold a very large domain. However, always consider how large your domain can grow especially with nested arrays
  • Atomic updates to documents are atomic per document. If you batch update several documents then each document will be updated atomically one by one but the whole operation is not atomic, i.e. we have no transactions
  • There’s no built-in mechanism to check for concurrent access to a document. The EntityFramework object context ensures that each operation is carried out in a thread-safe manner. There’s no equivalent of this context object in MongoDb so it’s possible that while you’re updating a document someone else deletes it. Or you change the rental fee to 2 and someone else updates it to 3 at the same time. The update to be processed last will win. However, this is nothing new – we could easily fill a whole textbook with concurrency issues in databases

In the next post we’ll look at how to delete documents.

View the posts related to data storage here.

MongoDB in .NET part 5: WriteConcerns and replacing documents

Introduction

In the previous part of this series we successfully inserted a new Car object in our database. We also managed to read all items from the database and show them in a table. We’ll continue looking at MongoDb operations.

We’ll keep working on the same demo as before so have it ready in Visual Studio.

WriteConcern

Recall that we inserted a new Car object via the MongoDb Insert method of the MongoCollection object. It returns a WriteConcernResult object that we haven’t bothered with. In the Create action method of the CarsController controller modify…

CarRentalContext.Cars.Insert(car);

…to…

WriteConcernResult writeResult = CarRentalContext.Cars.Insert(car);
bool ok = writeResult.Ok;

Insert a breakpoint within the action method, start the app, navigate to /cars/create and insert a new Car object. When the code execution stops inspect the “ok” boolean value in Visual Studio. It should be true, meaning that the write operation has succeeded. We could read this result because by default the driver will wait for an acknowledgement if some CRUD operation is carried out through the MongoClient object. There’s one more level to enhance durability – you can wait for a journal commit acknowledgement from the database. You can achieve this by either extending the connection string to…

mongodb://localhost/?journal=true

…or in code as follows:

String mongoHost = ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString;
MongoClientSettings settings =
	MongoClientSettings.FromUrl(new MongoUrl(mongoHost));	
settings.WriteConcern.Journal = true;
_mongoClient = new MongoClient(settings);

The WriteConcern property has at least one more important property you should know about: W. That’s right, a property called W. It is of type WValue and is represented by an integer. Say you have a cluster of MongoDb databases – a replica set – with one primary server for writes and 1 or more secondary servers for reads. The write operation will be propagated to all secondary servers in a matter of milliseconds. This is what you often see in real life database environments to enhance durability and data availability – if one server dies then you still have at least one more for reads and writes. The property W indicates the number of nodes in the cluster that must acknowledge the write – or update – operation. A similar mechanism from SQL Server is AlwaysOn. WriteConcern currently has 4 predefined values:

settings.WriteConcern = WriteConcern.W1;
settings.WriteConcern = WriteConcern.W2;
settings.WriteConcern = WriteConcern.W3;
settings.WriteConcern = WriteConcern.W4;

W1 means at least 1 node has to acknowledge the modification. You understand the rest. There is a special property where the majority of the nodes must acknowledge the operation:

settings.WriteConcern = WriteConcern.WMajority;

There’s another special value if you want to skip write acknowledgements altogether:

settings.WriteConcern = WriteConcern.Unacknowledged;

Update the CarRentalContext constructor to the following:

String mongoHost = ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString;
MongoClientSettings settings =
MongoClientSettings.FromUrl(new MongoUrl(mongoHost));
settings.WriteConcern = WriteConcern.Unacknowledged;
_mongoClient = new MongoClient(settings);			
_mongoServer = _mongoClient.GetServer();			
_mongoDatabase = _mongoServer.GetDatabase(ConfigurationManager.AppSettings["CarRentalDatabaseName"]);

Run the app and try to enter a new car. Press Create and then you should get a null pointer exception here:

bool ok = writeResult.Ok;

The writeResult object will be null as we didn’t want any acknowledgement. This is called fire-and-forget.

Why would you want to relax the acknowledgment mechanism? I can give you an example from my work. In one of our projects we monitor some real-time performance statistics during load testing of a web site and show it in various graphs. The process is very “insert” intensive with lots of inserts during the test. We don’t really care if a couple of data points are lost here and there as there are always new ones coming in. So to save time and get better performance we relax all types of write acknowledgment. Also, the stats are only interesting during the live testing so we’re not much concerned with durability either. You can even set the acknowledgement policy on the level of the operation, e.g.:

CarRentalContext.Cars.Insert(car, WriteConcern.Unacknowledged);

…or on the collection level:

public MongoCollection<Car> Cars
{
	get
	{
		return CarRentalDatabase.GetCollection<Car>("cars", WriteConcern.W1);
	}
}

However, for most types of data in a database you’ll want to turn on write acknowledgments. Change the CarRentalContext to the default write concern:

public CarRentalContext()
{
	String mongoHost = ConfigurationManager.ConnectionStrings["CarRentalConnectionString"].ConnectionString;
	MongoClientSettings settings =
		MongoClientSettings.FromUrl(new MongoUrl(mongoHost));			
	_mongoClient = new MongoClient(settings);			
	_mongoServer = _mongoClient.GetServer();			
	_mongoDatabase = _mongoServer.GetDatabase(ConfigurationManager.AppSettings["CarRentalDatabaseName"]);
}

Updates

Updates can be performed in 2 ways:

  • Replace an existing document with a new one
  • Modify an existing document without removing it

The Save() method will replace an existing document. Save() can also act as an Insert method: if there’s no document with the specified ID or no ID is provided at all then Save will create a new document. This is known as an upsert, i.e. update or insert. Save() returns a WriteConcernResult like Insert. There’s also another method called Update which Save calls upon internally. However, Save is more concise and saves you some typing.

The View for /cars created a table for us to show the cars. It has also prepared some links to update a record – check the Edit link. Don’t click on it yet as have no action method or controller for it.

In CarsController add the following Edit stub:

[HttpGet]
public ActionResult Edit(string id)
{

}

We’ll need to find the car with the selected ID and retrieve it from the database. To find an object by id you can use the FindOneById method on the collection like this:

[HttpGet]
public ActionResult Edit(string id)
{
      Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
}

You can build very sophisticated search criteria using the IMongoQuery builder. We can write the above query as follows:

IMongoQuery findByIdQuery = Query.EQ("_id", new ObjectId(id));
Car car = CarRentalContext.Cars.FindOne(findByIdQuery);

Query is used to build IMongoQuery objects. Just type “Query.” in VS and you’ll see lots of familiar properties such as “EQ” for equal, GT – greater than, LT – less than – and lots more. Whenever you need build a query to the Find or FindOne methods you’ll use the Query object. In case you need to specify a range of criteria you can combine them with And or Or:

Query.And(Query.EQ("someProperty", propertyValue), Query.EQ("otherProperty", otherValue));

Let’s continue. We’ll need a View for the Edit action and a corresponding view-model. Add a class called UpdateCarViewModel to the ViewModels folder:

public class UpdateCarViewModel
{
	[Editable(false)]
	public string Id { get; set; }
	[Required]
	[Display(Name = "Type of car")]
	public string Make { get; set; }
	[Required]
	[Display(Name = "Number of doors")]
	[Range(2, 6)]
	public int NumberOfDoors { get; set; }
	[Required]
	[Display(Name = "Daily rental fee")]
	public decimal DailyRentalFee { get; set; }
	[Required]
	[Display(Name = "List of allowed countries delimited with ';'")]
	public string DelimitedListOfCountries { get; set; }
}

It’s almost the same as the InsertCarViewModel, we could probably do with some subclassing, but it’s fine as it is for the demo. We’ll need the ID so that the Save method can locate the correct document. Next we’ll need to convert a Car object into an UpdateCarViewModel object. Insert the following extension method to DomainExtensions.cs:

public static UpdateCarViewModel ConvertToUpdateViewModel(this Car carDomain)
{
	UpdateCarViewModel updateVm = new UpdateCarViewModel()
	{
		Id = carDomain.Id
		, DailyRentalFee = carDomain.DailyRentalFee
		, Make = carDomain.Make
		, NumberOfDoors = carDomain.NumberOfDoors
	};

	if (carDomain.CountriesAllowedIn != null && carDomain.CountriesAllowedIn.Count() > 0)
	{
		StringBuilder sb = new StringBuilder();
		for (int i = 0; i < carDomain.CountriesAllowedIn.Count(); i++)
		{
			sb.Append(carDomain.CountriesAllowedIn.ElementAt(i));
			if (i < carDomain.CountriesAllowedIn.Count() - 1)
			{
				sb.Append(";");
			}
		}
		updateVm.DelimitedListOfCountries = sb.ToString();
	}

	return updateVm;
}

Back in CarsController modify the Edit GET method as follows:

[HttpGet]
public ActionResult Edit(string id)
{
	Car car = CarRentalContext.Cars.FindOneById(new ObjectId(id));
	return View(car.ConvertToUpdateViewModel());
}

Right-click “Edit”, select Add View:

Adding Edit template to Car object

Again, VS will create a basic Edit template for the selected item. Let’s check if it works so far. Run the application, navigate to /cars and click on the Edit link for one of the Car objects. This should open the following Edit form:

Edit Car template populated

Now we need to write the POST Edit action to complete the loop. Add the following stub to the CarsController:

[HttpPost]
public ActionResult Edit(UpdateCarViewModel updateCarViewModel)
{

}

We’ll need another extension method to convert an UpdateCarViewModel back to a Car. Add the following to DomainExtensions:

public static Car ConvertToDomain(this UpdateCarViewModel updateCarViewModel)
{
	Car car = new Car()
	{
		Id = updateCarViewModel.Id
		, DailyRentalFee = updateCarViewModel.DailyRentalFee
		, Make = updateCarViewModel.Make
		, NumberOfDoors = updateCarViewModel.NumberOfDoors
	};
	string[] countries = updateCarViewModel.DelimitedListOfCountries.Split(new char[] { ';' }, StringSplitOptions.RemoveEmptyEntries);
	car.CountriesAllowedIn = countries.ToList();
	return car;
}

Again, we’ll need the Id so that the Save function can find the document to be modified. Here’s the final version of the POST Edit method:

[HttpPost]
public ActionResult Edit(UpdateCarViewModel updateCarViewModel)
{
	if (ModelState.IsValid)
	{
		Car modifiedCar = updateCarViewModel.ConvertToDomain();
		CarRentalContext.Cars.Save(modifiedCar);
		return RedirectToAction("Index");
	}
	return View(updateCarViewModel);
}

We didn’t have to look up the Car object in the database – the Save method will check the ID for us. Run the application, update an existing Car and save it. If all works fine then you’ll be redirected to the Index page with the updated value in the list of Cars.

In the next post we’ll look at updates a bit closer.

View the posts related to data storage here.

MongoDB in .NET part 4: the POCO model, insertions, indexes and capped collections

Introduction

We saw the basics of POCO documents in the previous post of this series. In this part we’ll build up our model with POCOs.

Open the demo application CarRentalWeb we started out with and let’s get to work!

Our model

Add a new folder to the solution called Domain. In that folder insert a class called Car:

public class Car
{
        [BsonRepresentation(MongoDB.Bson.BsonType.ObjectId)]
	public string Id { get; set; }
	public string Make { get; set; }
	[BsonRepresentation(MongoDB.Bson.BsonType.Double)]
	public decimal DailyRentalFee { get; set; }
	public int NumberOfDoors { get; set; }
	public List<string> CountriesAllowedIn { get; set; }
}

You’ll recognise the BsonRepresentationAttribute from the previous post. You may be wondering why we have public string Id with the BsonRepresentation attribute. We saw in the previous post that Mongo stores the id field – called “_id” – in its own ObjectId format. We could simply write…

public ObjectId Id { get; set; }

…but ObjectId might cause problems with data binding elsewhere in the application as it is not a native .NET object. So we choose its nearest .NET approximation which is a string.

While we’re on this topic the above object definition may not even be considered strictly POCO anymore. It contains attributes that belong to a technology that a true domain object should not care about. These attributes have to do with data persistence and domain objects should have no knowledge of how they are represented in the data store. If you’re not sure what I mean I recommend that you go through the series on Domain Driven Design to learn how to separate domain and persistence models to keep the domain models strictly POCO. At the same time you may be fine with decorating your domain objects like that but it’s your decision.

However, this series is about MongoDb, not DDD, so we’ll allow for such deviations. You can then organise your objects the way you want in your project.

Inserting cars

As we said before a Mongo collection is a group of related documents. It is similar to a table in a relational database. Mongo collections are represented by the MongoCollection object in the C# driver and can be accessed via the MongoDatabase object. In our CarRentalContext context class we built such an object in the class constructor.

Add a new folder to the solution called ViewModels. ViewModels here are not the same as the behaviour-rich objects in MVVM. Instead they are simple DTOs – data transfer objects – to represent the “View” part of our models in MVC. Essentially they will be bound to the elements of the form where we insert new cars. Add the following class to the ViewModels folder:

public class InsertCarViewModel
{
	[Required]
	[Display(Name = "Type of car")]
	public string Make { get; set; }
	[Required]
	[Display(Name = "Number of doors")]
	[Range(2,6)]
	public int NumberOfDoors { get; set; }
	[Required]
	[Display(Name = "Daily rental fee")]
	public decimal DailyRentalFee { get; set; }
	[Required]
	[Display(Name = "List of allowed countries delimited with ';'")]
	public string DelimitedListOfCountries { get; set; }
}

This prepares some basic validation and display values using standard MVC annotations. Let’s prepare the Cars controller. Add a new empty MVC controller like this…:

Add cars controller in Visual Studio

Insert a Create action method like this:

[HttpGet]
public ActionResult Create()
{
	return View();
}

Right-click “Create” and select “Add View”. Fill in the form like this:

Create GET Create template for Car view model

In case the Model class list doesn’t show the InsertCarViewModel then close the window and build the project. Click OK and you’ll be presented an out-of-the box Create view. Start the application and navigate to /cars/create. You should see a simple form:

GET Create form to insert new cars

The VS 2013 equivalent will have a different style but that’s not important. When we press the Edit button the POST Create action will be called. Let’s insert it into the CarsController:

[HttpPost]
public ActionResult Create(InsertCarViewModel insertCarViewModel)
{
	if (ModelState.IsValid)
	{
	}
	return View(insertCarViewModel);
}

It’s not doing anything yet. We want to test if the incoming InsertCarViewModel object is correctly populated. Insert a breakpoint at the if statement. Start the application, navigate to /cars/create, fill in some valid values and press create. Code execution will stop at the breakpoint. Inspect the insertCarViewModel in VS – you’ll see that all properties have been populated correctly. We’ll stay on the same page after the Create method returns. The next step is to actually insert this record into MongoDb.

We’ll need to convert the viewmodel to its proper domain representation. We’ll use an extension method to do that. Insert a class called DomainExtensions in the Domain folder. Change the namespace to CarRentalWeb. Declare the class as static and insert the following static method in it:

namespace CarRentalWeb
{
	public static class DomainExtensions
	{
		public static Car ConvertToDomain(this InsertCarViewModel insertCarViewModel)
		{
			Car car = new Car()
			{
				DailyRentalFee = insertCarViewModel.DailyRentalFee
				, Make = insertCarViewModel.Make
				, NumberOfDoors = insertCarViewModel.NumberOfDoors
			};
			string[] countries = insertCarViewModel.DelimitedListOfCountries.Split(new char[] { ';' }, StringSplitOptions.RemoveEmptyEntries);
			car.CountriesAllowedIn = countries.ToList();
			return car;
		}
	}
}

Back in Create(InsertCarViewModel insertCarViewModel) call this extension method as follows:

if (ModelState.IsValid)
{
	Car car = insertCarViewModel.ConvertToDomain();
}

We’ll need access to the database context in this controller – and most likely in all our controllers. Add a new class called BaseController to the Controllers folder. Select the “class” template, not the New Controller options.

public class BaseController : Controller
{
	private CarRentalContext _carRentalContext;

	public BaseController()
	{
		_carRentalContext = new CarRentalContext();
	}

	public CarRentalContext CarRentalContext
	{
		get
		{
			return _carRentalContext;
		}
	}
}

Change the declaration of CarsController to inherit from the BaseController:

public class CarsController : BaseController

Next we’ll extend the CarRentalContext class to return a mongo collection that holds the Cars objects. Open CarRentalContext.cs and add the following getter:

public MongoCollection<Car> Cars
{
	get
	{
		return CarRentalDatabase.GetCollection<Car>("cars");
	}
}

The MongoCollection class has an overloaded version where you can specify the type of objects stored in the collection. The MongoDatabase object is used to get hold of a collection of a specific type. We need to specify the name of the collection – “cars” – which is an arbitrary string that you can specify, it could as well be “MickeyMouse”, but you probably prefer something descriptive. The type specification will help the serialisation mechanism to convert to and from Car and its BSON representation.

Back in CarsController.Create we’ll use the simplest approach to insert a new document. Add the following line after the conversion from the view model:

CarRentalContext.Cars.Insert(car);
return RedirectToAction("Index");

That’s as simple as it gets. We’ll come back to the Insert method in the next post as it returns a WriteConcern object which deserves more attention. We can ignore it for the time being.

After an insertion we’d like to return to the Index page which lists all the Car objects in a table, which is why we have the RedirectToAction call. The CarsController already has an Index method which returns a View(). Let’s tie the View to an IEnumerable of Car view-model objects. We don’t have those yet so add a new class called CarViewModel to the ViewModels folder:

public class CarViewModel
{	
        [Display(Name = "ID")]
	public string Id { get; set; }
	public string Make { get; set; }
	[Display(Name = "Rental fee per day")]
	public decimal DailyRentalFee { get; set; }
	[Display(Name = "Number of doors")]
	public int NumberOfDoors { get; set; }
	[Display(Name = "Allowed countries")]
	public string CountriesAllowedIn { get; set; }
}

We’ll need to convert the Car objects from the database into the CarViewModel representations. Add the following extension methods to DomainExtensions.cs.:

public static IEnumerable<CarViewModel> ConvertAllToViewModels(this IEnumerable<Car> carDomains)
{
	foreach (Car car in carDomains)
	{
		yield return car.ConvertToViewModel();
	}
}

public static CarViewModel ConvertToViewModel(this Car carDomain)
{
	CarViewModel carViewModel = new CarViewModel()
	{
		Id = carDomain.Id
		, DailyRentalFee = carDomain.DailyRentalFee
		, Make = carDomain.Make
		, NumberOfDoors = carDomain.NumberOfDoors
	};

	if (carDomain.CountriesAllowedIn != null && carDomain.CountriesAllowedIn.Count() > 0)
	{
		StringBuilder sb = new StringBuilder();
		for (int i = 0; i < carDomain.CountriesAllowedIn.Count(); i++)
		{
			sb.Append(carDomain.CountriesAllowedIn.ElementAt(i));
			if (i < carDomain.CountriesAllowedIn.Count() - 1)
			{
				sb.Append(",");
			}
		}
		carViewModel.CountriesAllowedIn = sb.ToString();
	}
        return carViewModel;
}

The CarsController Index action will need to fetch all the car objects, convert them to view models and show them to the viewer. Compile the solution and right-click “Index”. Select Add View…:

Add view to list all car objects

This will give you a basic template to show all items in the sequence in a table.

We can use the FindAll method of MongoCollection to retrieve all items without any search parameters:

public ActionResult Index()
{
	List<Car> carsInDb = CarRentalContext.Cars.FindAll().ToList();
        return View(carsInDb.ConvertAllToViewModels());
}

We should be good to go. Start the application, navigate to /cars/create. Fill in the form with valid data, e.g.:

Create car form values example

If everything’s OK then you’ll see the new Car object listed in its CarViewModel representation on the Index page:

Car created and index page shown

That wasn’t too difficult, right? We didn’t even need to create a database or collection. They were both created for us upon the first insert.

Indexes

You’ll probably know that it’s important to set an index on fields that often figure in searches. Here’s how you can set a composite index on the Car object on the Make and NumberOfDoors properties – which probably doesn’t make sense, this is only example code:

IndexKeysBuilder<Car> carIndexBuilder = IndexKeys<Car>.Ascending(c => c.Make, c => c.NumberOfDoors);
IndexOptionsBuilder<Car> carIndexOptions = IndexOptions<Car>.SetName("Car_CompositeIndex").SetTimeToLive(new TimeSpan(2, 0, 0, 0));
CarRentalContext.Cars.EnsureIndex(carIndexBuilder, carIndexOptions);

With SetTimeToLive you can set an expiry date on a collection.

You can drop an Index by its name as follows:

CarRentalContext.Cars.DropIndexByName("Car_CompositeIndex");

Capped collections

A capped collection is a collection which is not allowed to grow beyond a specific size in bytes. If the maximum size is reached then the elements will be dropped starting from the first element in a FIFO fashion. Here’s how you can set the cap to 50MB:

CollectionOptionsBuilder optionsBuilder = new CollectionOptionsBuilder();
optionsBuilder.SetCapped(true);
optionsBuilder.SetMaxSize(52428800);
CarRentalContext.CarRentalDatabase.CreateCollection("NewCollection", optionsBuilder);

Read the next part of the series here.

View the posts related to data storage here.

ultimatemindsettoday

A great WordPress.com site

Elliot Balynn's Blog

A directory of wonderful thoughts

HarsH ReaLiTy

A Good Blog is Hard to Find

Softwarearchitektur in der Praxis

Wissenswertes zu Webentwicklung, Domain-Driven Design und Microservices

Technology Talks

on Microsoft technologies, Web, Android and others

Software Engineering

Web development

Disparate Opinions

Various tidbits

chsakell's Blog

WEB APPLICATION DEVELOPMENT TUTORIALS WITH OPEN-SOURCE PROJECTS

Once Upon a Camayoc

Bite-size insight on Cyber Security for the not too technical.

Guru N Guns's

OneSolution To dOTnET.

Johnny Zraiby

Measuring programming progress by lines of code is like measuring aircraft building progress by weight.

%d bloggers like this: