Using Amazon DynamoDb with the AWS .NET API Part 6: queries

Introduction

In the previous post we looked at how to delete and update records in Amazon DynamoDb. In this post we’ll investigate how to run queries against the data set in a DynamoDb table. Like in the two posts before this we’ll look at the document and data models separately.

We’ll extend the DynamoDbDemo console application in Visual Studio.

I’ve inserted a couple more records into the demo DynamoDb table in preparation for the queries:

Read more of this post

Using Amazon DynamoDb with the AWS .NET API Part 5: updating and deleting records

Introduction

In the previous post we saw how to insert records into an Amazon DynamoDb table. We looked at two distinct programmatic ways to build our objects to be inserted: the loosely typed Document model and the strongly typed, more object oriented Data model.

In this post we’ll see how to update and delete records using the two models.

As a reminder we inserted the following records into our DynamoDb People table in the previous post:

Read more of this post

Using Amazon DynamoDb with the AWS .NET API Part 4: record insertion

Introduction

In the previous post we looked at table-related operations in Amazon DynamoDb: table creation, deletion and update. In this post we’ll continue our discussion of DynamoDb by insertions. We’ll see that there are two ways to build up the records and insert them into DynamoDb.

Open the demo application we’ve been working on in Visual Studio and let’s start.

Preparation

For this post we’ll need to recreate the test table we deleted at the end of the previous post. Make the following call through Main to create it anew under a different name:

DynamoDbDemoService service = new DynamoDbDemoService();
service.CreateNewTableDemo("People");

Wait until the table becomes active.

Inserting records into a DynamoDb table

There are two distinct ways to represent a record that should be inserted into DynamoDb through the AWS .NET API: the Document model and the Data model.

The Document model follows the traditional NoSql approach of loose coupling and loose typing. The properties of the object – which will be the record in the table – are described using strings. This is the model to follow in case you’d like to have the freedom to enter any record type into the table. Well, almost any record type, as you’ll need to provide values for the key(s) at a minimum.

The Data model follows OOP and strong typing. The properties of the record are described in proper classes and using DynamoDb specific attributes. This is the model to follow in case you want to apply a strict and predicable schema to your records.

Document model insertion

Let’s look at the document model first. Insert the following code into DynamoDbDemoService:

public void InsertPeopleByDocumentModel(string tableName)
{
	try
	{
		using (IAmazonDynamoDB client = GetDynamoDbClient())
		{
			Table peopleTable = Table.LoadTable(client, tableName);
			Document firstPerson = new Document();
			firstPerson["Name"] = "John";
			firstPerson["Birthdate"] = new DateTime(1980, 06, 24);
			firstPerson["Address"] = "34 Wide Street, London, UK";
			firstPerson["Age"] = 34;
			firstPerson["Neighbours"] = new List<String>() { "Jane", "Samantha", "Richard" };
			peopleTable.PutItem(firstPerson);

			Document secondPerson = new Document();
			secondPerson["Name"] = "Jill";
			secondPerson["Birthdate"] = new DateTime(1981, 02, 26);
			secondPerson["Address"] = "52 Broad Street, Dublin, Ireland";
			secondPerson["Age"] = 33;
			secondPerson["Neighbours"] = new List<String>() { "Alex", "Greg", "Michael" };
			peopleTable.PutItem(secondPerson);

			Document thirdPerson = new Document();
			thirdPerson["Name"] = "George";
			thirdPerson["Birthdate"] = new DateTime(1979, 11, 4);
			thirdPerson["Address"] = "118 Main Street, Washington";
			thirdPerson["Age"] = 35;
			thirdPerson["Neighbours"] = new List<String>() { "Kathrine", "Kate", "Christine" };
			peopleTable.PutItem(thirdPerson);

			Document fourthPerson = new Document();
			fourthPerson["Name"] = "Carole";
			fourthPerson["Birthdate"] = new DateTime(1984, 4, 10);
			fourthPerson["Address"] = "5 Short Street, Sydney, Australia";
			fourthPerson["Age"] = 30;
			fourthPerson["Neighbours"] = new List<String>() { "Nadia", "Katya", "Malcolm" };
			peopleTable.PutItem(fourthPerson);
		}
	}
	catch (AmazonDynamoDBException exception)
	{
		Debug.WriteLine(string.Concat("Exception while inserting records into DynamoDb table: {0}", exception.Message));
		Debug.WriteLine(String.Concat("Error code: {0}, error type: {1}", exception.ErrorCode, exception.ErrorType));
	}
}

We load the table and then build up 4 objects using a dictionary in the Document object. The objects describe 4 people that have some properties like Age and Address in common. You see that we use the general Document object and not some specialised Person object to do this. For each new record we call the PutItem method of the Table object.

Call the method from Main as follows:

DynamoDbDemoService service = new DynamoDbDemoService();
service.InsertPeopleByDocumentModel("People");

Highlight table in DynamoDb and click Explore table:

The records should be visible in our DynamoDb People table:

You’ll see that the birth dates look a bit funny as even the hour-minute-seconds sections were filled in, but it’s OK for now. The lists of neighbours were automatically converted into Json-like strings.

Delete all items from DynamoDb before we look at the Data model:

Data model

Here we’ll represent the loosely typed objects above as “proper” classes. Each Person object will have an Address object as well which will be formatted using Json.

Add a NuGet referencet to a JSON framework as Address object will be serialised and deserialised using the popular JSON.NET library:

Let’s insert the Address class first:

public class Address
{
	public Address()
	{
		Street = "N/A";
		City = "N/A";
		Country = "N/A";
	}

	public string Street { get; set; }
	public string City { get; set; }
	public string Country { get; set; }
}

Each Person object will have an Address but the address must be represented in DynamoDb in one of the accepted data types: number, string or binary. The most straightforward solution is to convert an Address object into a Json string. The AWS .NET library has a special interface for that called IPropertyConverter which requires you to implement to methods: FromEntry and ToEntry. They help you describe how an object should be serialised and deserialised.

Insert the following address converter into the solution:

public class AddressConverter : IPropertyConverter
{
	public object FromEntry(DynamoDBEntry entry)
	{
		Primitive primitive = entry as Primitive;
		if (primitive == null) return new Address();

		if (primitive.Type != DynamoDBEntryType.String)
		{
			throw new InvalidCastException(string.Format("Address cannot be converted as its type is {0} with a value of {1}"
				, primitive.Type, primitive.Value));
		}

		string json = primitive.AsString();
		return JsonConvert.DeserializeObject<Address>(json);
	}

	public DynamoDBEntry ToEntry(object value)
	{
		Address address = value as Address;
		if (address == null) return null;

		string json = JsonConvert.SerializeObject(address);
		return new Primitive(json);
	}
}

In FromEntry we check if the incoming DynamoDb entry is null and if it’s of type string. We can only convert strings into Address objects. We finally deserialise the JSON string into an Address object and return it. In ToEntry we do the opposite and and convert the incoming object first into an Address and then serialise it to its JSON representation.

We can now build the Person object:

[DynamoDBTable("People")]
public class Person
{
	[DynamoDBHashKey]
	public string Name { get; set; }
	[DynamoDBRangeKey(AttributeName="Birthdate")]
	public DateTime BirthDate { get; set; }
	[DynamoDBProperty(Converter = typeof(AddressConverter))]
	public Address Address { get; set; }

	public int Age { get; set; }
	public List<string> Neighbours { get; set; }
}

You can see how we use DynamoDb attributes to describe our hash and range keys. We also provide the converter for Address. You can use the DynamoDBProperty attribute to indicate in case the property name in DynamoDb is different from the property name in your class. This can be useful in case you follow a different naming structure in DynamoDb. Here’s an example:

[DynamoDBProperty(AttributeName="helloThisIsTheAge")]
public int Age { get; set; }

We can now insert the equivalent of the InsertPeopleByDocumentModel method using the Data model. Add the following InsertPeopleByDataModel method to DynamoDbDemoService:

[DynamoDBTable("People")]
public void InsertPeopleByDataModel(string tableName)
{
	try
	{
		using (IAmazonDynamoDB client = GetDynamoDbClient())
		{
			DynamoDBContext context = new DynamoDBContext(client);
			Person firstPerson = new Person()
			{
				Name = "John"
				, BirthDate = new DateTime(1980, 06, 24)
				, Address = new Address() { Street = "34 Wide Street", City = "London", Country = "UK" }
				, Age = 34
				, Neighbours = new List<String>() { "Jane", "Samantha", "Richard" }
			};

			Person secondPerson = new Person()
			{
				Name = "Jill"
				, BirthDate = new DateTime(1981, 02, 26)
				, Address = new Address() { Street = "52 Broad Street", City = "Dublin", Country = "Ireland" }
				, Age = 33
				, Neighbours = new List<String>() { "Alex", "Greg", "Michael" }
			};

			Person thirdPerson = new Person()
			{
				Name = "George"
				, BirthDate = new DateTime(1979, 11, 4)
				, Address = new Address() { Street = "118 Main Street", City = "Washington", Country = "USA" }
				, Age = 35
				, Neighbours = new List<String>() { "Kathrine", "Kate", "Christine" }
			};

			Person fourthPerson = new Person()
			{
				Name = "Carole"
				, BirthDate = new DateTime(1984, 4, 10)
				, Address = new Address() { Street = "5 Short Street", City = "Sydney", Country = "Australia" }
				, Age = 30
				, Neighbours = new List<String>() { "Nadia", "Katya", "Malcolm" }
			};

			context.Save<Person>(firstPerson);
			context.Save<Person>(secondPerson);
			context.Save<Person>(thirdPerson);
			context.Save<Person>(fourthPerson);
		}
	}
	catch (AmazonDynamoDBException exception)
	{
		Debug.WriteLine(string.Concat("Exception while inserting records into DynamoDb table: {0}", exception.Message));
		Debug.WriteLine(String.Concat("Error code: {0}, error type: {1}", exception.ErrorCode, exception.ErrorType));
	}
}

We build up the same 4 objects as before but now as proper Person objects. We finally use the DynamoDBContext object to save each new Person.

Call the method from Main:

service.InsertPeopleByDataModel("People");

You’ll find the records in our DynamoDb table:

Note how the Address is shown as Json.

In the next post we’ll see how to update and delete records using both models.

View all posts related to Amazon Web Services and Big Data here.

Using Amazon DynamoDb with the AWS.NET API Part 3: table operations

Introduction

In the previous post we started coding our DynamoDb demo application. In particular we saw how to extract information about tables in DynamoDb.

In this post we’ll look at various table operations, more specifically how to create, delete and update tables in DynamoDb.

Open the demo application we started working on previously and let’s get to it!

Creating tables

We can set the properties of our table through the CreateTableRequest object. This is where we set the table name, the provisioned read and write throughput, the key schema and their attribute definitions, and the secondary indexes. The following code sets all of these except the secondary keys.

Insert the following method into DynamoDbDemoService.cs:

public void CreateNewTableDemo(string tableName)
{
	try
	{
		using (IAmazonDynamoDB client = GetDynamoDbClient())
		{				
			CreateTableRequest createTableRequest = new CreateTableRequest();
			createTableRequest.TableName = tableName;
			createTableRequest.ProvisionedThroughput = new ProvisionedThroughput() { ReadCapacityUnits = 1, WriteCapacityUnits = 1 };
			createTableRequest.KeySchema = new List<KeySchemaElement>()
			{
				new KeySchemaElement()
				{
					AttributeName = "Name"
					, KeyType = KeyType.HASH
				},
				new KeySchemaElement()
				{
					AttributeName = "Birthdate"
					, KeyType = KeyType.RANGE
				}
			};
			createTableRequest.AttributeDefinitions = new List<AttributeDefinition>()
			{
				new AttributeDefinition(){AttributeName = "Name", AttributeType = ScalarAttributeType.S}
				, new AttributeDefinition(){AttributeName = "Birthdate", AttributeType = ScalarAttributeType.S}
			};
			CreateTableResponse createTableResponse = client.CreateTable(createTableRequest);

			TableDescription tableDescription = createTableResponse.TableDescription;
			Debug.WriteLine(string.Format("Table {0} creation command sent to Amazon. Current table status: {1}", tableName, tableDescription.TableStatus));

			String tableStatus = tableDescription.TableStatus.Value.ToLower();
			while (tableStatus != "active")
			{
				Debug.WriteLine(string.Format("Table {0} not yet active, waiting...", tableName));
				Thread.Sleep(2000);
				DescribeTableRequest describeTableRequest = new DescribeTableRequest(tableName);
				DescribeTableResponse describeTableResponse = client.DescribeTable(describeTableRequest);
				tableDescription = describeTableResponse.Table;
				tableStatus = tableDescription.TableStatus.Value.ToLower();
				Debug.WriteLine(string.Format("Latest status of table {0}: {1}", tableName, tableStatus));
			}

			Debug.WriteLine(string.Format("Table creation loop exited for table {0}, final status: {1}", tableName, tableStatus));
		}
	}
	catch (AmazonDynamoDBException exception)
	{
		Debug.WriteLine(string.Concat("Exception while creating new DynamoDb table: {0}", exception.Message));
		Debug.WriteLine(String.Concat("Error code: {0}, error type: {1}", exception.ErrorCode, exception.ErrorType));
	}
}

You’ll recognise the Hash and Range key types from the first post of this series. The KeySchemaElement object lets you define the name and type of the key. The AttributeDefinition will further let you define the schema of the keys.

We set up a compound key that consists of “Name” and “Birthdate”. We’ll insert some people-related objects in the next post where each record will have a compound key of Name and Birthdate, i.e. the name and birthday properties will uniquely identify each record. You’ll note the attribute type of S in the AttributeDefinition object. S stands for String. We saw earlier that DynamoDb supports 3 data types: strings, numbers and binary data. Numbers are represented by ScalarAttributeType.N and binary types by ScalarAttributeType.B. As dates are not supported by DynamoDb we’ll go with strings instead.

After constructing the CreateTableRequest object we call the CreateTable method of the DynamoDb client. The CreateTableResponse includes a TableDescription object which helps us read the initial status of the table. It should be “creating” at first. We then enter a loop where we continuously check the status of the new table until it reaches status “active”.

Let’s call the method from Main as follows:

DynamoDbDemoService service = new DynamoDbDemoService();
service.CreateNewTableDemo("a-demo-table");

If all goes well then you should see output similar to the following:

Table a-demo-table creation command sent to Amazon. Current table status: CREATING
Table a-demo-table not yet active, waiting…
Latest status of table a-demo-table: creating
Table a-demo-table not yet active, waiting…
Latest status of table a-demo-table: creating
Table a-demo-table not yet active, waiting…
Latest status of table a-demo-table: active
Table creation loop exited for table a-demo-table, final status: active

The table is up and running in the DynamoDB GUI as well:

It’s obviously not allowed to have two tables with the same name. Try running the same code again and you should get an exception message:

Exception while creating new DynamoDb table: {0}Table already exists: a-demo-table
Error code: {0}, error type: {1}ResourceInUseExceptionUnknown

Updating a table

You can update a table through the UpdateTableRequest object. You can update the read and write throughput values and the global secondary indexes, nothing else. The following method demonstrates the usage of the object:

public void UpdateTableDemo(string tableName)
{
	try
	{
		using (IAmazonDynamoDB client = GetDynamoDbClient())
		{
			UpdateTableRequest updateTableRequest = new UpdateTableRequest();
			updateTableRequest.TableName = tableName;
			updateTableRequest.ProvisionedThroughput = new ProvisionedThroughput() { ReadCapacityUnits = 2, WriteCapacityUnits = 2 };					
			UpdateTableResponse updateTableResponse = client.UpdateTable(updateTableRequest);
			TableDescription tableDescription = updateTableResponse.TableDescription;
			Debug.WriteLine(string.Format("Update table command sent to Amazon for table {0}, status after update: {1}", tableName
				, tableDescription.TableStatus));
		}			
	}
	catch (AmazonDynamoDBException exception)
	{
		Debug.WriteLine(string.Concat("Exception while updating DynamoDb table: {0}", exception.Message));
		Debug.WriteLine(String.Concat("Error code: {0}, error type: {1}", exception.ErrorCode, exception.ErrorType));
	}
}

Run the method like this:

DynamoDbDemoService service = new DynamoDbDemoService();
//service.UpdateTableDemo("a-demo-table");

Here’s the Debug output:

Update table command sent to Amazon for table a-demo-table, status after update: UPDATING

…and the table has been updated with the new throughput values:

Deleting a table

Deleting a table is as trivial as providing the name of the table to the DeleteTableRequest object and sending it to AWS by way of the DynamoDb client:

public void DeleteTableDemo(string tableName)
{
	try
	{
		using (IAmazonDynamoDB client = GetDynamoDbClient())
		{
			DeleteTableRequest deleteTableRequest = new DeleteTableRequest(tableName);
			DeleteTableResponse deleteTableResponse = client.DeleteTable(deleteTableRequest);
			TableDescription tableDescription = deleteTableResponse.TableDescription;
			TableStatus tableStatus = tableDescription.TableStatus;
			Debug.WriteLine(string.Format("Delete table command sent to Amazon for table {0}, status after deletion: {1}", tableName
				, tableDescription.TableStatus));
		}
	}
	catch (AmazonDynamoDBException exception)
	{
		Debug.WriteLine(string.Concat("Exception while deleting DynamoDb table: {0}", exception.Message));
		Debug.WriteLine(String.Concat("Error code: {0}, error type: {1}", exception.ErrorCode, exception.ErrorType));
	}
}

Run the method as follows:

DynamoDbDemoService service = new DynamoDbDemoService();
service.DeleteTableDemo("a-demo-table");

The table won’t be deleted immediately but have the status “deleting” at first before the table is finally deleted:

Delete table command sent to Amazon for table a-demo-table, status after deletion: DELETING

Here’s what it looks like in the DynamoDb GUI:

We’ll explore how to insert records into a DynamoDb table in the next post.

View all posts related to Amazon Web Services and Big Data here.