Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 12: querying the geolocation range to DynamoDb

Introduction

In the previous post we created the DynamoDb source file with a reduced set of geolocations from the full MaxMind data source. We saw how we could reuse the same process as before in the case of the IP and coordinate range tables.

In this final post of this series we’ll close the loop by actually extracting and geographic properties of a geoname ID. After all you’d like to know whether a visitor come from New York other than “geoname ID 3452334”.

Read more of this post

Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 11: uploading the geolocation range to DynamoDb

Introduction

In the previous post we successfully queried the coordinate range database in DynamoDb. We used the query endpoints that are built into the AWS geo-location library to find the data records within the radius around a centre point.

Where are we now?

We’ve got quite far with our project. We have the ability to query an IPv4 and coordinate range table in DynamoDb. We can extract a geoname ID that belongs to either an IP or a latitude-longitude pair. The next step is to dress up those IDs with real location data such as “Stockholm” or “Tehran”.

Read more of this post

Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 10: querying the coordinate range table

Introduction

In the previous post we loaded the limited lng/lat range records into DynamoDb. As we’re only talking about about 50 records we could have added them in code one by one. However, that strategy would never work for the full MaxMind data set even after discarding the duplicates. So instead we looked at the built-in Import/Export functionality in DynamoDb. You’ll be able to go through the same process when you’re ready to import the full data set.

In this post we’ll see how to query the lnglat range database to extract the ID of the nearest geolocation. We’ll get to use the AWS Java SDK.

Read more of this post

Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 9: uploading the co-ordinate range to DynamoDb

Introduction

In the previous post we successfully created the lng/lat import file that DynamoDb can understand and process.

In this post we’ll upload this file to DynamoDb. The process will be the same to what we saw in this post where we inserted the demo data into the IPv4 range table. If necessary then re-read that post to refresh your memory about the process. We’ll follow the strategy we laid out in this post.

Read more of this post

Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 8: creating the lng/lat coordinates source file for DynamoDb

Introduction

In the previous post we successfully queried the limited IPv4 range table in DynamoDb and found the geoname ID that belongs to a single IP. We used 3 available integer properties in the table to narrow down the number of records that had to be scanned to reduce the query execution time and the risk for exceptions.

In this post we’ll start the same process for the lng/lat coordinate range. More specifically we’ll prepare the raw data file that can be uploaded into DynamoDb through S3. The process will be very similar to what we saw in this post where we created the IPv4 range source file. It is good idea to quickly re-scan that post to remind you of the process.

Read more of this post

Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 7: querying the IPv4 range table

Introduction

In the previous post we loaded the limited IP range records into DynamoDb. As we’re only talking about about 250 records we could have added them in code one by one. However, that strategy would never work for the full MaxMind IP data set of 10 million records. So instead we looked at the built-in Import/Export functionality in DynamoDb. You’ll be able to go through the same process when you’re ready to import the full data set.

In this post we’ll see how to query the IP range database to extract the ID of the nearest geolocation. We’ll get to use the AWS Java SDK.

Read more of this post

Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 6: uploading IPv4 range to DynamoDb

Introduction

In the previous post we successfully created a limited IPv4 range file ready to be uploaded to DynamoDb. We saw how the relevant bits were extracted from the reduced subset of the MaxMind CSV source file and how the DynamoDb-specific input file was created.

In this post we’ll see how to upload the source file to DynamoDb using the bulk insertion tools available there. We’ll only import our limited test data but the same steps apply for large data sets as well.

Read more of this post

Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 5: creating the IPv4 source file for DynamoDb

Introduction

In the previous post we went through our strategy to save the longitude-latitude coordinates in DynamoDb for our geo-spatial queries later on. We said that we would save the records in DynamoDb in a way so that it fits queries according to a library designed by AWS which in turn uses a geo-library from Google.

In this post we’ll finally see some action. We’re ready to format an upload the IP range to DynamoDb. Actually we’ll show the techniques using only a small subset of the MaxMind raw data source. I strongly recommend you follow the same strategy and not try to upload 10 million rows at once. Make sure the process works for a small subset from start to finish and then go for the real thing. The steps outlined in this series will also apply to the full, paid version of the CSV source.

Read more of this post

Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 4: lng/lat range strategy

Introduction

In the previous post we discussed our strategy to save the IP ranges in DynamoDb. We saw that it would be very inefficient to store the IPs as strings and run our queries based on some string manipulation. Instead we’ll store the IP ranges as lower and upper limit integers.

In this post we’ll discuss our strategy to save the longitude-latitude ranges for cities.

Read more of this post

Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 3: IPv4 range strategy

Introduction

In the previous post we went through the details of the CSV source files that show the IP and lng/lat ranges and the actual locations. We saw that the two source files are linked by the location ID.

The next task is to import the source into DynamoDb. Recall that we want to handle queries based on IPs and lng/lat pairs separately, those are the primary goals of this series. The way to query an IP database is very different from querying a lng/lat database. An IP will fit into some IP range and we’d like to find that record. Whereas if you have a lng/lat co-ordinate pair and would like to find the nearest city/school/hospital/etc. within a certain radius then that query will involve some complex maths instead.

Read more of this post