Using Amazon DynamoDb with the AWS.NET API Part 1: introduction

Introduction

The usage of NoSql database solutions has been increasing rapidly in recent years. If you’re not familiar with NoSql then here’s a definition from Wikipedia:

“A NoSQL database provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases.”

If you used document-based databases such as MongoDb or RavenDb then this is straightforward. If not, then I have an introductory course on MongoDb with .NET starting here. Skim through the first couple of posts there to get the basic idea.

As Amazon Web Services has a solution for all imaginable software engineering problem – well, almost all of them – it’s no surprise that they have a NoSql database as well. Their NoSql database is called DynamoDb. It is a fast, scalable and efficient NoSql storage that can act as the data store of any application type that can communicate through HTTP. Amazon have prepared a wide range of SDKs and .NET is no exception. Therefore any .NET application will be able to communicate with DynamoDb: creating and querying tables, adding, deleting and updating records will be straightforward.

You’ll need access to Amazon Web Services in order to try the examples. DynamoDb has a free-tier meaning you can play around in it with some limited data. That’s more than enough for evaluation purposes. This page includes more information on how free-tier works and how to set up an account. By signing up with Amazon and creating a user you’ll also get a pair of security keys: an Amazon Access Key and a Secret Access Key. You can also create users with the Identity and Access Management (IAM) tool of Amazon.

The home page includes a lot of marketing stuff but there’s a lot documentation available for developers as well starting here. Beware that most examples are available in Java.

Goals of this series

This series is the 3rd installment of a larger set of series based around Big Data in Amazon Cloud. Previously we discussed the message handler solution called Amazon Kinesis and an efficient blob storage mechanism called S3. The whole purpose of the set of series – or series of series – is to go through the basics of a Big Data system in Amazon cloud. At the same time, each ingredient in the series should be straightforward to follow for those who are not interested in Big Data, but only in the Amazon component itself.

This sub-series is no different. If you only want to learn about DynamoDb then you can follow through without worrying much about what we discussed before. If, on the other hand, you’re building a Big Data handling system then we’ll see towards the end of the series where DynamoDb could fit into the picture. We’ll pave the way for the next Amazon Big Data component called Elastic MapReduce (EMR), a Hadoop-based data mining solution. In fact, I was a bit uncertain whether I should describe EMR first and then go over to DynamoDb, but showing DynamoDb first should be fine.

The DynamoDb UI

Log onto the AWS console and locate the DynamoDb service:

Probably every service you use with AWS has a region that you can select in the top right section of the UI:

Note that in your case a different region might be pre-selected so don’t get confused by the fact that it says “Singapore” in the above screenshot. Click the down-pointing arrow to view all the available regions. These regions are significant for all services with a couple of exceptions. E.g. S3, which we discussed in the previous series, is global and has less regional significance. In the case of DynamoDb when you create a new database then that database will be available in the selected region. It doesn’t, however, mean that users cannot save and access records in a database in Ireland from Australia. However, it will take Australian users a bit more time to complete the DB-related operations than it does for a user in the UK. Also, we’ll see later that the region must be specified in code when configuring the access to AWS otherwise you may be wondering why your database cannot be located.

If this is the first time you use DynamoDb then you’ll probably see the below button to create your first table:

Else if you already have at least one table in the selected region then you’ll see the table-handling toolbar above the list of tables:

In any case press the Create Table button and you’ll see the following popup window:

You’ll understand the Table Name text box. Primary Key, however, deserves more attention. It offers two options: “Hash and Range” and “Range”. This page describes their purpose. Here’s a summary:

• Hash and range: this is comparable to composite IDs in MS SQL where the primary key consists of two columns that together uniquely identify a record
• Hash: a single unique primary key

In both cases you are responsible for making the IDs not-null and unique. If you in code try to enter a record whose ID already exists then that row will be updated instead with the incoming properties.

The popup also gives the choice of data type: string, number or binary. In fact, these are the three data types that are supported in DynamoDb. There are no separate int, bigint, decimal etc. types, Number takes care of all. Also, there’s no native DateTime type. If you need to store dates then you can store them as UNIX timestamps as a Number data type or as a formatted string as a String data type, e.g. “2014-04-15 13:43:32”.

So, enter a table name, like “a-first-test”, a primary key named “Id” of type Hash/String and click Continue. The next screen will show the options to create one or more index – don’t worry about them now, click continue. Enter 5 as write capacity unit and 10 as read capacity – these are the upper limits for the free tier. Capacity units determine the read and write throughput for your DynamoDb tables and Amazon will set up resources to meet the required throughput. The free tier limits may change of course so don’t forget to check exactly – if you hover your mouse above the “free tier” link then the small popup will show you the actual values:

Click continue.

On the next screen you’ll be able to set up alarms in the case of those read and write units being close to the limit. Don’t worry about that now so uncheck the Use Basic Alarms option and click Continue.

The next screen will summarise your choices. Click create. The table will be first in CREATING status:

Refresh the table and eventually it will be created:

You can also delete a table through the Delete Table button, but let’s keep this table for the time being. We’ll extract some information about it in code in the next post.

This exercise was enough to see what DynamoDb looks like. We haven’t loaded any records yet. You’ve probably noticed that we haven’t specified any column names and data types like we normally do in relational databases such as MS SQL Server. This is no surprise to those familiar with NoSql databases where column names and data types can vary from record to record.

This is enough for starters. We’ll start looking at some code in the next post.

View all posts related to Amazon Web Services and Big Data here.