Using Amazon S3 with the AWS.NET API Part 4: working with folders in code

Introduction

In the previous post we looked at some more basic code examples to work with Amazon S3. In the first part we saw how to create folders within a bucket in the S3 GUI. We haven’t yet seen how to create and delete folders in code and that’s the goal of this post.

We’ll extend our demo application AmazonS3Demo so have it open in Visual Studio 2012/2013.

Creating folders

So far we’ve seen that it’s fairly simple and self-explanatory to work with objects in S3: PutObjectRequest, ListObjectsRequest etc. were easy to handle. However, there’s no special C# object for folders in the SDK, like PutFolderRequest. The PutObjectRequest is used to create folders but they need to be used in a special way.

Locate S3DemoService.cs in the demo app and add the following method:

public void RunFolderCreationDemo()
{
	using (IAmazonS3 s3Client = GetAmazonS3Client())
	{
		try
		{					
			PutObjectRequest folderRequest = new PutObjectRequest();					
			String delimiter = "/";
			folderRequest.BucketName = "a-second-bucket-test";
			String folderKey = string.Concat("this-is-a-subfolder", delimiter);
			folderRequest.Key = folderKey;
			folderRequest.InputStream = new MemoryStream(new byte[0]);
			PutObjectResponse folderResponse = s3Client.PutObject(folderRequest);
                }
		catch (AmazonS3Exception e)
		{
			Console.WriteLine("Folder creation has failed.");
			Console.WriteLine("Amazon error code: {0}",
				string.IsNullOrEmpty(e.ErrorCode) ? "None" : e.ErrorCode);
			Console.WriteLine("Exception message: {0}", e.Message);
		}
	}
}

Most of the code looks probably familiar from the RunFileUploadDemo() method. The way to indicate that we’re intending to create a demo is to attach a “/” delimiter to the key name and set the content input stream to an empty byte array. Run this code from Main…:

static void Main(string[] args)
{
	S3DemoService demoService = new S3DemoService();
	demoService.RunFolderCreationDemo();

	Console.WriteLine("Main done...");
	Console.ReadKey();
}

…and the folder should be visible in S3:

What if you’d like to add another folder within this folder? It’s the same code as above, you just need to attach the name of the subfolder and the delimiter to the object key:

String folderKey = string.Concat("this-is-a-subfolder", delimiter, "another-subfolder",delimiter);
					folderRequest.Key = folderKey;

…and there it is:

Checking if a folder exists

Say you’d like to find out whether the “this-is-a-subfolder/another-subfolder/” folder path exists. The following code snippet will first check if “this-is-a-subfolder” exists:

public void RunFolderExistenceCheckDemo()
{
	using (IAmazonS3 s3Client = GetAmazonS3Client())
	{
		try
		{
			ListObjectsRequest findFolderRequest = new ListObjectsRequest();
			findFolderRequest.BucketName = "a-second-bucket-test";
			findFolderRequest.Delimiter = "/";
			findFolderRequest.Prefix = "this-is-a-subfolder";
			ListObjectsResponse findFolderResponse = s3Client.ListObjects(findFolderRequest);
			List<String> commonPrefixes = findFolderResponse.CommonPrefixes;
			Boolean folderExists = commonPrefixes.Any();
		}
		catch (AmazonS3Exception e)
		{
			Console.WriteLine("Folder existence check has failed.");
			Console.WriteLine("Amazon error code: {0}",
				string.IsNullOrEmpty(e.ErrorCode) ? "None" : e.ErrorCode);
			Console.WriteLine("Exception message: {0}", e.Message);
		}
	}
}

This is similar to what we had in RunObjectListingDemo. However, we don’t check for the existence of file objects but so-called prefixes. Folder names will be part of the full path to the object and the folder names will be the prefixes in that full path. If the CommonPrefixes string list is empty then the folder doesn’t exist.

Running this code will result in “true” for the folderExists variable.

Checking whether the subfolder within the folder exists requires just a minor change:

findFolderRequest.Prefix = "this-is-a-subfolder/another-subfolder";

Again, folderExists will be true.

Adding files to folders

Let’s add a file to the folders. The following code will insert c:\logfile.txt to “this-is-a-subfolder”:

public void RunObjectInsertionToFolderDemo()
{
	FileInfo filename = new FileInfo(@"c:\logfile.txt");
	string contents = File.ReadAllText(filename.FullName);
	using (IAmazonS3 s3Client = GetAmazonS3Client())
	{
		try
		{
			PutObjectRequest putObjectRequest = new PutObjectRequest();
			putObjectRequest.ContentBody = contents;
			String delimiter = "/";
			putObjectRequest.BucketName = string.Concat("a-second-bucket-test", delimiter, "this-is-a-subfolder"); 					
			putObjectRequest.Key = filename.Name;
			PutObjectResponse putObjectResponse = s3Client.PutObject(putObjectRequest);
		}
		catch (AmazonS3Exception e)
		{
			Console.WriteLine("File creation within folder has failed.");
			Console.WriteLine("Amazon error code: {0}",
				string.IsNullOrEmpty(e.ErrorCode) ? "None" : e.ErrorCode);
			Console.WriteLine("Exception message: {0}", e.Message);
		}
	}
}

The only difference to the RunFileUploadDemo() method we saw before is that we need to extend the bucket name with the delimiter and the folder name, i.e. the prefix. Here’s the file in S3:

You can probably guess how to upload the file to “this-is-a-subfolder/another-subfolder/”:

putObjectRequest.BucketName = string.Concat("a-second-bucket-test", delimiter, "this-is-a-subfolder", delimiter, "another-subfolder"); 					

…and here it is:

List objects in a folder

Listing file objects within a folder is a bit tricky as even the folders themselves are objects so they will be returned by the following query:

ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
String delimiter = "/";
listObjectsRequest.BucketName = "a-second-bucket-test";
listObjectsRequest.Prefix = string.Concat("this-is-a-subfolder", delimiter, "another-subfolder");				    
ListObjectsResponse listObjectsResponse = s3Client.ListObjects(listObjectsRequest);
foreach (S3Object entry in listObjectsResponse.S3Objects)
{
	if (entry.Size > 0)
	{
		Console.WriteLine("Found object with key {0}, size {1}, last modification date {2}", entry.Key, entry.Size, entry.LastModified);
	}
}					

The S3Objects list will hold 2 objects:

this-is-a-subfolder/another-subfolder/ of size 0
this-is-a-subfolder/another-subfolder/logfile.txt of size greater than 0

Hence we test for the size being greater than 0 to find “real” files. The output will be similar to the following:

Found object with key this-is-a-subfolder/another-subfolder/logfile.txt, size 44
90, last modification date 2014-12-09 22:26:39

Deleting folders

The following method will attempt to delete the this-is-a-subfolder/another-subfolder/ folder:

public void RunFolderDeletionDemo()
{
	using (IAmazonS3 s3Client = GetAmazonS3Client())
	{
		try
		{
			DeleteObjectRequest deleteFolderRequest = new DeleteObjectRequest();
			deleteFolderRequest.BucketName = "a-second-bucket-test";
			String delimiter = "/";
			deleteFolderRequest.Key = string.Concat("this-is-a-subfolder", delimiter, "another-subfolder", delimiter);
			DeleteObjectResponse deleteObjectResponse = s3Client.DeleteObject(deleteFolderRequest);					
		}
		catch (AmazonS3Exception e)
		{
			Console.WriteLine("Folder deletion has failed.");
			Console.WriteLine("Amazon error code: {0}",
				string.IsNullOrEmpty(e.ErrorCode) ? "None" : e.ErrorCode);
			Console.WriteLine("Exception message: {0}", e.Message);
		}
	}
}							

If you run this code then seemingly everything’s gone fine, there are no exceptions thrown. However, the folder is still visible in the UI. The reason is that at present there’s a file within the folder and folders must not be deleted if they are not empty. So in order for the above code to succeed we need to first check it has any objects in it using ListObjectsRequest code snippet above. Then all those elements must be deleted. Only then can the folder itself be deleted.

In the next post we’ll connect all this to our previous demo on Amazon Kinesis where we want to save the incoming URL response time observations in S3 in an organised manner.

View all posts related to Amazon Web Services and Big Data here.