Using Amazon S3 with the AWS.NET API Part 2: code basics

Introduction

In the previous post we looked at an overview of Amazon S3 and we also tried a couple of simple operations on the GUI. In this post we’ll start coding. If you followed along the previous series on Amazon Kinesis then the setup section will be familiar to you. Otherwise I’ll assume that you are starting this series without having read anything else on this blog.

Note that we’ll be concentrating on showing and explaining the technical code examples related to AWS. We’ll ignore software principles like SOLID and layering so that we can stay focused. It’s your responsibility to organise your code properly. There are numerous posts on this blog that take up topics related to software architecture.

Installing the SDK

The Amazon .NET SDK is available through NuGet. Open Visual Studio 2012/2013 and create a new C# console application called AmazonS3Demo. The purpose of this application will be to demonstrate the different parts of the SDK around S3. In reality the S3 handler could be any type of application:

• A website
• A Windows/Android/iOS app
• A Windows service
• etc.

…i.e. any application that’s capable of sending HTTP/S requests to a web service endpoint. We’ll keep it simple and not waste time with view-related tasks.

Install the following NuGet package:

Preparations

We cannot just call the services within the AWS SDK without proper authentication. This is an important reference page to handle your credentials in a safe way. We’ll the take the recommended approach and create a profile in the SDK Store and reference it from app.config.

This series is not about AWS authentication so we won’t go into temporary credentials but later on you may be interested in that option too. Since we’re programmers and it takes a single line of code to set up a profile we’ll go with the programmatic options. Add the following line to Main:

Amazon.Util.ProfileManager.RegisterProfile("demo-aws-profile", "your access key id", "your secret access key");

I suggest you remove the code from the application later on in case you want to distribute it. Run the application and it should execute without exceptions. Next open app.config and add the appSettings section with the following elements:

<appSettings>
        <add key="AWSProfileName" value="demo-aws-profile"/>
</appSettings>

First demo: listing the available buckets

We’ll put all our test code into a separate class. Insert a cs file called S3DemoService. We’ll need a method to build a handle to the service which is of type IAmazonS3:

private IAmazonS3 GetAmazonS3Client()
{
	return Amazon.AWSClientFactory.CreateAmazonS3Client(RegionEndpoint.EUWest1);
}

Note that we didn’t need to provide our credentials here. They will be extracted automatically using the profile name in the config file.

Let’s first find out what buckets we have in S3. This is almost a trivial task:

public void RunBucketListingDemo()
{
	using (IAmazonS3 s3Client = GetAmazonS3Client())
	{
		try
		{
			ListBucketsResponse response = s3Client.ListBuckets();
			List<S3Bucket> buckets = response.Buckets;
			foreach (S3Bucket bucket in buckets)
			{
				Console.WriteLine("Found bucket name {0} created at {1}", bucket.BucketName, bucket.CreationDate);
			}
		}
		catch (AmazonS3Exception e)
		{
			Console.WriteLine("Bucket listing has failed.");
			Console.WriteLine("Amazon error code: {0}",
				string.IsNullOrEmpty(e.ErrorCode) ? "None" : e.ErrorCode );
			Console.WriteLine("Exception message: {0}", e.Message);
		}
	}
}

We use the client to list all the available buckets. The method returns a ListBucketsResponse object which will hold the buckets.

You’ll see a lot of these Request and Response objects throughout the AWS SDK. Amazon are fond of wrapping the request parameters and response properties into Request and Response objects adhering to the RequestResponse pattern.

We then list the name and creation date of our buckets. Call this method from Main:

static void Main(string[] args)
{
	S3DemoService demoService = new S3DemoService();
	demoService.RunBucketListingDemo();

	Console.WriteLine("Main done...");
	Console.ReadKey();
}

Example output:

Found bucket name a-first-bucket-test created at 2014-12-03 22:30:59

Second demo: creating a new bucket

Creating a new bucket is equally easy. Add the following method to S3DemoService:

public void RunBucketCreationDemo()
{
	using (IAmazonS3 s3Client = GetAmazonS3Client())
	{
		try
		{
			PutBucketRequest putBucketRequest = new PutBucketRequest();
			String newBucketName = "a-second-bucket-test";
			putBucketRequest.BucketName = newBucketName;
			PutBucketResponse putBucketResponse = s3Client.PutBucket(putBucketRequest);					
		}
		catch (AmazonS3Exception e)
		{
			Console.WriteLine("Bucket creation has failed.");
			Console.WriteLine("Amazon error code: {0}",
				string.IsNullOrEmpty(e.ErrorCode) ? "None" : e.ErrorCode);
			Console.WriteLine("Exception message: {0}", e.Message);
		}
	}
}

Again, we can see a Request and a corresponding Response object to create a bucket. We only specify a name which means that we go with the default values for e.g. permissions, they are usually fine. PutBucketRequest provides some properties to indicate values not adhering to the default ones. E.g. here’s how to give Everyone the permission to view the bucket:

S3Grant grant = new S3Grant();
S3Permission permission = new S3Permission("List");
S3Grantee grantee = new S3Grantee();
grantee.CanonicalUser = "Everyone";
grant.Grantee = grantee;
grant.Permission = permission;
List<S3Grant> grants = new List<S3Grant>() { grant };
putBucketRequest.Grants = grants;

Call RunBucketCreationDemo from Main as follows:

demoService.RunBucketCreationDemo();

Run the application and the bucket should be created:

It’s not allowed to have two buckets with the same name. Run the application again and you should get an exception:

Bucket creation has failed.
Amazon error code: BucketAlreadyOwnedByYou
Exception message: Your previous request to create the named bucket succeeded and you already own it.

Third demo: file creation

Let’s upload a text file to the bucket we’ve just created. To be exact, we’ll upload its contents. Create some text file on your hard drive such as c:\logfile.txt and add some text to it. This example shows how to set the contents of the request. We also set an arbitrary metadata key-value pair of key “type” and value “log”. You can set any type of metadata you want:

public void RunFileUploadDemo()
{
	FileInfo filename = new FileInfo( @"c:\logfile.txt");
	string contents = File.ReadAllText(filename.FullName);
	using (IAmazonS3 s3Client = GetAmazonS3Client())
	{
		try
		{
			PutObjectRequest putObjectRequest = new PutObjectRequest();
			putObjectRequest.ContentBody = contents;
			putObjectRequest.BucketName = "a-second-bucket-test";
			putObjectRequest.Metadata.Add("type", "log");
			putObjectRequest.Key = filename.Name;
			PutObjectResponse putObjectResponse = s3Client.PutObject(putObjectRequest);
		}
		catch (AmazonS3Exception e)
		{
			Console.WriteLine("File creation has failed.");
			Console.WriteLine("Amazon error code: {0}",
				string.IsNullOrEmpty(e.ErrorCode) ? "None" : e.ErrorCode);
			Console.WriteLine("Exception message: {0}", e.Message);
		}
	}
}

Call this method from Main:

demoService.RunFileUploadDemo();

Run the application and the file with its custom metadata should be visible in S3:

We’ll continue with more examples in the next post.

View all posts related to Amazon Web Services and Big Data here.