Using Amazon S3 with the AWS.NET API Part 4: working with folders in code

Introduction

In the previous post we looked at some more basic code examples to work with Amazon S3. In the first part we saw how to create folders within a bucket in the S3 GUI. We haven’t yet seen how to create and delete folders in code and that’s the goal of this post.

We’ll extend our demo application AmazonS3Demo so have it open in Visual Studio 2012/2013.

Creating folders

So far we’ve seen that it’s fairly simple and self-explanatory to work with objects in S3: PutObjectRequest, ListObjectsRequest etc. were easy to handle. However, there’s no special C# object for folders in the SDK, like PutFolderRequest. The PutObjectRequest is used to create folders but they need to be used in a special way.

Locate S3DemoService.cs in the demo app and add the following method:

public void RunFolderCreationDemo()
{
	using (IAmazonS3 s3Client = GetAmazonS3Client())
	{
		try
		{					
			PutObjectRequest folderRequest = new PutObjectRequest();					
			String delimiter = "/";
			folderRequest.BucketName = "a-second-bucket-test";
			String folderKey = string.Concat("this-is-a-subfolder", delimiter);
			folderRequest.Key = folderKey;
			folderRequest.InputStream = new MemoryStream(new byte[0]);
			PutObjectResponse folderResponse = s3Client.PutObject(folderRequest);
                }
		catch (AmazonS3Exception e)
		{
			Console.WriteLine("Folder creation has failed.");
			Console.WriteLine("Amazon error code: {0}",
				string.IsNullOrEmpty(e.ErrorCode) ? "None" : e.ErrorCode);
			Console.WriteLine("Exception message: {0}", e.Message);
		}
	}
}

Most of the code looks probably familiar from the RunFileUploadDemo() method. The way to indicate that we’re intending to create a demo is to attach a “/” delimiter to the key name and set the content input stream to an empty byte array. Run this code from Main…:

static void Main(string[] args)
{
	S3DemoService demoService = new S3DemoService();
	demoService.RunFolderCreationDemo();

	Console.WriteLine("Main done...");
	Console.ReadKey();
}

…and the folder should be visible in S3:

Folder created in Amazon S3 bucket

What if you’d like to add another folder within this folder? It’s the same code as above, you just need to attach the name of the subfolder and the delimiter to the object key:

String folderKey = string.Concat("this-is-a-subfolder", delimiter, "another-subfolder",delimiter);
					folderRequest.Key = folderKey;

…and there it is:

Folder within a folder in Amazon S3 bucket

Checking if a folder exists

Say you’d like to find out whether the “this-is-a-subfolder/another-subfolder/” folder path exists. The following code snippet will first check if “this-is-a-subfolder” exists:

public void RunFolderExistenceCheckDemo()
{
	using (IAmazonS3 s3Client = GetAmazonS3Client())
	{
		try
		{
			ListObjectsRequest findFolderRequest = new ListObjectsRequest();
			findFolderRequest.BucketName = "a-second-bucket-test";
			findFolderRequest.Delimiter = "/";
			findFolderRequest.Prefix = "this-is-a-subfolder";
			ListObjectsResponse findFolderResponse = s3Client.ListObjects(findFolderRequest);
			List<String> commonPrefixes = findFolderResponse.CommonPrefixes;
			Boolean folderExists = commonPrefixes.Any();
		}
		catch (AmazonS3Exception e)
		{
			Console.WriteLine("Folder existence check has failed.");
			Console.WriteLine("Amazon error code: {0}",
				string.IsNullOrEmpty(e.ErrorCode) ? "None" : e.ErrorCode);
			Console.WriteLine("Exception message: {0}", e.Message);
		}
	}
}

This is similar to what we had in RunObjectListingDemo. However, we don’t check for the existence of file objects but so-called prefixes. Folder names will be part of the full path to the object and the folder names will be the prefixes in that full path. If the CommonPrefixes string list is empty then the folder doesn’t exist.

Running this code will result in “true” for the folderExists variable.

Checking whether the subfolder within the folder exists requires just a minor change:

findFolderRequest.Prefix = "this-is-a-subfolder/another-subfolder";

Again, folderExists will be true.

Adding files to folders

Let’s add a file to the folders. The following code will insert c:\logfile.txt to “this-is-a-subfolder”:

public void RunObjectInsertionToFolderDemo()
{
	FileInfo filename = new FileInfo(@"c:\logfile.txt");
	string contents = File.ReadAllText(filename.FullName);
	using (IAmazonS3 s3Client = GetAmazonS3Client())
	{
		try
		{
			PutObjectRequest putObjectRequest = new PutObjectRequest();
			putObjectRequest.ContentBody = contents;
			String delimiter = "/";
			putObjectRequest.BucketName = string.Concat("a-second-bucket-test", delimiter, "this-is-a-subfolder"); 					
			putObjectRequest.Key = filename.Name;
			PutObjectResponse putObjectResponse = s3Client.PutObject(putObjectRequest);
		}
		catch (AmazonS3Exception e)
		{
			Console.WriteLine("File creation within folder has failed.");
			Console.WriteLine("Amazon error code: {0}",
				string.IsNullOrEmpty(e.ErrorCode) ? "None" : e.ErrorCode);
			Console.WriteLine("Exception message: {0}", e.Message);
		}
	}
}

The only difference to the RunFileUploadDemo() method we saw before is that we need to extend the bucket name with the delimiter and the folder name, i.e. the prefix. Here’s the file in S3:

File within a folder in Amazon S3 bucket

You can probably guess how to upload the file to “this-is-a-subfolder/another-subfolder/”:

putObjectRequest.BucketName = string.Concat("a-second-bucket-test", delimiter, "this-is-a-subfolder", delimiter, "another-subfolder"); 					

…and here it is:

File within a folder within a folder in Amazon S3 bucket

List objects in a folder

Listing file objects within a folder is a bit tricky as even the folders themselves are objects so they will be returned by the following query:

ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
String delimiter = "/";
listObjectsRequest.BucketName = "a-second-bucket-test";
listObjectsRequest.Prefix = string.Concat("this-is-a-subfolder", delimiter, "another-subfolder");				    
ListObjectsResponse listObjectsResponse = s3Client.ListObjects(listObjectsRequest);
foreach (S3Object entry in listObjectsResponse.S3Objects)
{
	if (entry.Size > 0)
	{
		Console.WriteLine("Found object with key {0}, size {1}, last modification date {2}", entry.Key, entry.Size, entry.LastModified);
	}
}					

The S3Objects list will hold 2 objects:

this-is-a-subfolder/another-subfolder/ of size 0
this-is-a-subfolder/another-subfolder/logfile.txt of size greater than 0

Hence we test for the size being greater than 0 to find “real” files. The output will be similar to the following:

Found object with key this-is-a-subfolder/another-subfolder/logfile.txt, size 44
90, last modification date 2014-12-09 22:26:39

Deleting folders

The following method will attempt to delete the this-is-a-subfolder/another-subfolder/ folder:

public void RunFolderDeletionDemo()
{
	using (IAmazonS3 s3Client = GetAmazonS3Client())
	{
		try
		{
			DeleteObjectRequest deleteFolderRequest = new DeleteObjectRequest();
			deleteFolderRequest.BucketName = "a-second-bucket-test";
			String delimiter = "/";
			deleteFolderRequest.Key = string.Concat("this-is-a-subfolder", delimiter, "another-subfolder", delimiter);
			DeleteObjectResponse deleteObjectResponse = s3Client.DeleteObject(deleteFolderRequest);					
		}
		catch (AmazonS3Exception e)
		{
			Console.WriteLine("Folder deletion has failed.");
			Console.WriteLine("Amazon error code: {0}",
				string.IsNullOrEmpty(e.ErrorCode) ? "None" : e.ErrorCode);
			Console.WriteLine("Exception message: {0}", e.Message);
		}
	}
}							

If you run this code then seemingly everything’s gone fine, there are no exceptions thrown. However, the folder is still visible in the UI. The reason is that at present there’s a file within the folder and folders must not be deleted if they are not empty. So in order for the above code to succeed we need to first check it has any objects in it using ListObjectsRequest code snippet above. Then all those elements must be deleted. Only then can the folder itself be deleted.

In the next post we’ll connect all this to our previous demo on Amazon Kinesis where we want to save the incoming URL response time observations in S3 in an organised manner.

View all posts related to Amazon Web Services and Big Data here.

Advertisements

About Andras Nemes
I'm a .NET/Java developer living and working in Stockholm, Sweden.

6 Responses to Using Amazon S3 with the AWS.NET API Part 4: working with folders in code

  1. Pingback: Using Amazon S3 with the AWS.NET API | Alejandro Clemotte

  2. Karthikeyan R says:

    I am using bellow code to write text into “test.txt”. it writting text “message 1″ and again i try to write second time like”message 2” it overwrite the old text “message 1” to “message 2”. i need not to overwrite old text.

    PutObjectRequest request = new PutObjectRequest();
    request.BucketName = BUCKET_NAME;
    request.Key = “test.txt”;
    request.ContentBody = “this is exsisting messages 2”;
    client.PutObject(request);

    • Andras Nemes says:

      Hello, you cannot update an existing document in S3 unfortunately. The only roundabout I know of is to first check if the document exists, then download its content, append the new text and upload the revised document. //Andras

      • Karthikeyan R says:

        Thank you very much Andras Nemes for noticing my query, i am going to synchronize the data from local to s3 bucket.

  3. Karthikeyan R says:

    Hi Andras Nemes:
    i need to read excel file from s3 bucket. i used the bellow method but it return some encode format i could not convert into input stream format in C#.

    public string Get_JsonString_From_AmazonS3_Bucket(string S3_Key, string BucketName)
    {
    string S3BucketJsonString = “”;
    try
    {
    GetObjectRequest GetRequest = new GetObjectRequest();
    GetRequest.BucketName = BucketName;
    GetRequest.Key = S3_Key;
    GetObjectResponse GetResponse = client.GetObject(GetRequest);
    using (StreamReader sReader = new StreamReader(GetResponse.ResponseStream))
    {
    S3BucketJsonString = sReader.ReadToEnd();
    }
    return S3BucketJsonString;
    }
    catch (Exception ex)
    {
    return null;
    throw (ex);
    }
    }

    • Andras Nemes says:

      Hello, I’m not sure, MS Excel files may have some special formatting. Anyhow, this doesn’t sound like an S3 problem. Have you tried to convert an Excel file on your file system into an input stream?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

ultimatemindsettoday

A great WordPress.com site

Elliot Balynn's Blog

A directory of wonderful thoughts

Robin Sedlaczek's Blog

Developer on Microsoft Technologies

HarsH ReaLiTy

My goal with this blog is to offend everyone in the world at least once with my words… so no one has a reason to have a heightened sense of themselves. We are all ignorant, we are all found wanting, we are all bad people sometimes.

Softwarearchitektur in der Praxis

Wissenswertes zu Webentwicklung, Domain-Driven Design und Microservices

the software architecture

thoughts, ideas, diagrams,enterprise code, design pattern , solution designs

Technology Talks

on Microsoft technologies, Web, Android and others

Software Engineering

Web development

Disparate Opinions

Various tidbits

chsakell's Blog

Anything around ASP.NET MVC,WEB API, WCF, Entity Framework & AngularJS

Cyber Matters

Bite-size insight on Cyber Security for the not too technical.

Guru N Guns's

OneSolution To dOTnET.

Johnny Zraiby

Measuring programming progress by lines of code is like measuring aircraft building progress by weight.

%d bloggers like this: