How to manage Amazon Machine Images with the .NET Amazon SDK Part 2: monitoring and terminating AMI instances, managing Security Groups

In the previous post we successfully sent a launch request to the selected AMI. We’ll now see how to monitor its status and terminate it.

Open up the Console application we worked on previously. We finished off where the user selected an AMI and we sent a launch request to EC2 in order to get one instance running of that AMI. The method that retrieves the status of the machine looks as follows:

private static string RetrieveInstanceStatus(string instanceId, Region selectedRegion)
{
	AmazonEC2Client amazonEc2client = GetAmazonClient(selectedRegion.Endpoint);
	try
	{
		DescribeInstancesRequest instancesRequest = new DescribeInstancesRequest();
		Filter filter = new Filter();
		filter.Name = "instance-id";
		filter.Value = new List<string>() { instanceId };
		instancesRequest.Filter = new List<Filter>() { filter };
		DescribeInstancesResponse instancesResponse = amazonEc2client.DescribeInstances(instancesRequest);
		DescribeInstancesResult instancesResult = instancesResponse.DescribeInstancesResult;
		Reservation reservations = instancesResult.Reservation[0];
		RunningInstance runningInstance = reservations.RunningInstance[0];
		return runningInstance.InstanceState.Name;
	}
	catch
	{
		throw;
	}
}

Most of the code will look familiar from the previous post. We send in the selected region and the ID of the instance of the selected AMI. Remember that we requested to start up exactly one instance of the AMI. When you launch that instance then the instance will get a unique id which is a property of the RunningInstance object. The LaunchImage method returned a list of RunningInstance objects where we’ll find that ID, we’ll get to that in a second. Back in the above method we’ll set a filter to the DescribeInstancesRequest object as we’re only interested in that very instance. We don’t care about the status of other instances. Again, as we know that we only launched one instance it’s OK to return the first element in the Reservation and RunningInstance collections which we get back from the DescribeInstancesResult object.

A short aside: it’s perfectly feasible to start multiple instances of the same image. You’ll need to set the MinCount and MaxCount properties of the RunInstancesRequest object accordingly. Take a look at the LaunchImage method we implemented earlier. This returns a list of RunningInstance objects that you can use to collect all the individual instance IDs. The instance ID list can be sent into a slightly modified RetrieveInstanceStatus method which accepts a list of instance ids instead of just one instance id as in this specific implementation. The filter value of the DescribeInstancesRequest will then be set to the list of IDs and you’ll get back the status of all instances.

Let’s add one more helper method to Program.cs that loops until the image instance has reached the “running” state:

private static void MonitorInstanceStartup(string instanceId, Region selectedRegion)
{
	string status = "N/A";
	while (status != "running")
	{
		status = RetrieveInstanceStatus(instanceId, selectedRegion);
		Console.WriteLine(string.Format("Current status of instance {0}: {1}", instanceId, status));
		Thread.Sleep(1000);
	}
}

So we simply wait for the machine to reach the “running” state. The extended Main method looks like this:

static void Main(string[] args)
{
	List<Region> amazonRegions = GetAmazonRegions();
	PrintAmazonRegions(amazonRegions);
	int usersChoice = GetSelectedRegionOfUser(amazonRegions);
	Region selectedRegion = amazonRegions[usersChoice - 1];
	List<Amazon.EC2.Model.Image> imagesInRegion = GetSuitableImages(selectedRegion);
	PrintAmis(imagesInRegion);
	int usersImageChoice = GetSelectedImageOfUser(imagesInRegion);
	Image selectedImage = imagesInRegion[usersImageChoice - 1];
	List<RunningInstance> launchedInstances = LaunchImage(selectedImage, selectedRegion);
	MonitorInstanceStartup(launchedInstances[0].InstanceId, selectedRegion);

	Console.ReadKey();
}

Let’s run the app. If everything goes well then you should see an output similar to this:

I’ll check in the EC2 management window as well:

A word of caution: although the state of the machine is running it really should say ‘initialising’ at first and then running. You’ll notice that it doesn’t take long to reach the running state, maybe 10-15 seconds. However, the instance may not reach a truly usable “running” state until 2-3 more minutes. “Running” can be compared to the first blue screen on a Windows machine where it says “Starting Windows”. That is not really running yet, right? And then the startup process runs, extra applications and processes are loaded etc. and when all that’s done then you can start working on your machine normally.

Let’s see how we can terminate the instance:

private static Tuple<string, string> TerminateInstance(string instanceId, Region selectedRegion)
{
	AmazonEC2Client amazonEc2client = GetAmazonClient(selectedRegion.Endpoint);
	try
	{
		TerminateInstancesRequest terminateRequest = new TerminateInstancesRequest();
		terminateRequest.InstanceId = new List<string>() { instanceId };
		TerminateInstancesResponse terminateResponse = amazonEc2client.TerminateInstances(terminateRequest);
		TerminateInstancesResult terminateResult = terminateResponse.TerminateInstancesResult;
		List<InstanceStateChange> stateChanges = terminateResult.TerminatingInstance;
		return new Tuple<string, string>(stateChanges[0].CurrentState.Name, stateChanges[0].PreviousState.Name);
	}
	catch
	{
		throw;
	}
}

As usual we set the current region in the Amazon client. Then we send a TerminateInstancesRequest whose purpose is quite self-explanatory I believe. You can terminate multiple instances by sending in a list of instance ids. In our case it’s a list containing one element only. We get back a list of InstanceStateChange object where we can read among other things the current state of the Instance and the state that it had just before the termination request was made.

Run the application and you may see an output similar to the following:

So you see that the “running” state changes to “shutting-down” after the termination request was issued. Let’s also monitor the shutting-down phase until the instance is fully terminated:

private static void MonitorInstanceShutdown(string instanceId, Region selectedRegion)
{
	string status = "N/A";
	while (status != "terminated")
	{
		status = RetrieveInstanceStatus(instanceId, selectedRegion);
		Console.WriteLine(string.Format("Current status of instance {0}: {1}", instanceId, status));
		Thread.Sleep(1000);
	}
}

Add the following to Main:

MonitorInstanceShutdown(launchedInstances[0].InstanceId, selectedRegion);

Run the application and you may see something similar to this:

Let’s check in the EC2 manager as well just to make sure it worked:

Security groups

A security group is a firewall to control the access to the instances. You can read about it on the AWS website here. You can control Security Groups programmatically.

Use the following code to search for a certain security group by name:

private static void SearchSecurityGroup(Region selectedRegion)
{
	DescribeSecurityGroupsRequest securityGroupRequest = new DescribeSecurityGroupsRequest();
	Filter groupNameFilter = new Filter();
	groupNameFilter.Name = "group-name";
	groupNameFilter.Value = new List<String>() { "Security group name" };
	List<Filter> securityGroupRequestFilter = new List<Filter>();
	securityGroupRequestFilter.Add(groupNameFilter);
	securityGroupRequest.Filter = securityGroupRequestFilter;
	DescribeSecurityGroupsResponse securityGroupResponse = GetAmazonClient(selectedRegion.Endpoint).DescribeSecurityGroups(securityGroupRequest);
	DescribeSecurityGroupsResult securityGroupResult = securityGroupResponse.DescribeSecurityGroupsResult;
	List<SecurityGroup> securityGroups = securityGroupResult.SecurityGroup;
}

The code follows the AWS SDK style we’ve seen so far: construct a Request object, set a Filter on it, send the request to the selected region and read the result from the Response. I encourage you to inspect the SecurityGroup object to see what properties can be extracted from it.

You can inspect the Ip permissions of the selected security group as follows:

private static void InspectIpPermissions(SecurityGroup selectedSecurityGroup)
{
	List<IpPermission> ipPermissions = selectedSecurityGroup.IpPermission;
	foreach (IpPermission ipPermission in ipPermissions)
	{
		StringBuilder ipRangeBuilder = new StringBuilder();
		foreach (String ipRange in ipPermission.IpRange)
		{
			ipRangeBuilder.Append(ipRange).Append(", ");
		}
		Console.WriteLine(string.Format("Protocol: {0}, from port: {1}, to port: {2}, ip range: {3}", ipPermission.IpProtocol	, ipPermission.FromPort, ipPermission.ToPort, ipRangeBuilder.ToString()));
	}
}

You can extract the IP and port ranges and some other properties of the IpPermission object.

The following method creates a new Security Group and opens up port HTTP and HTTPS traffic for all incoming IPs on the TCP protocol:

private static void CreateSecurityGroup(Region selectedRegion)
{
	CreateSecurityGroupRequest createGroupRequest = new CreateSecurityGroupRequest();
	createGroupRequest.GroupName = "Security group name";
	createGroupRequest.GroupDescription = "Security group description";
	AmazonEC2Client amazonEc2Client = GetAmazonClient(selectedRegion.Endpoint);
	amazonEc2Client.CreateSecurityGroup(createGroupRequest);
	int[] ports = { 80, 443 };
	foreach (int i in ports)
	{
		AuthorizeSecurityGroupIngressRequest ingressRequest = new AuthorizeSecurityGroupIngressRequest();
		ingressRequest.GroupName = "Security group name";
		ingressRequest.IpProtocol = "tcp";
		ingressRequest.FromPort = i;
		ingressRequest.ToPort = i;
		ingressRequest.CidrIp = "0.0.0.0/0";
		amazonEc2Client.AuthorizeSecurityGroupIngress(ingressRequest);
	}
}

You can also remove Security Groups using the following code:

private static void DeleteSecurityGroup(Region selectedRegion)
{
	AmazonEC2Client amazonEc2Client = GetAmazonClient(selectedRegion.Endpoint);
	DeleteSecurityGroupRequest deleteGroupRequest = new DeleteSecurityGroupRequest();
	deleteGroupRequest.GroupName = "Group name to be deleted";
	DeleteSecurityGroupResponse deleteGroupResponse = amazonEc2Client.DeleteSecurityGroup(deleteGroupRequest);
	
}

How to manage Amazon Machine Images with the .NET Amazon SDK Part 1: starting an image instance

If you have an access to Amazon Web Services (AWS) EC2 then you can manage Amazon Machine Images (AMI) in the cloud using this screen:

In case you are not familiar with AMIs then here‘s a short summary.

Amazon has created SDKs in for several different programming languages, such as Java, Python, Ruby, C# etc. by which you can manage the servers, AMIs etc. in an elegant way in code. You can check out the various packages on the AWS homepage:

In this post I’ll concentrate on the .NET package in a console application: how to start, monitor and shut down the AMI instances. Note that if you don’t have an Amazon account then it will be difficult to test the provided code samples. You will need both your Access Key ID and your Secret Access Key in order to communicate with AWS through the SDK.

Demo

In the demo I’ll concentrate on showing the functionality and ignore best practices such as SOLID, IoC containers, patterns, DRY etc. in order to eliminate the “noise”. I’ll put all code within Program.cs. It’s up to you how you organise it in your code later.

Open Visual Studio 2012 and create a new Console application. Add a reference to the AWS SDK using NuGet:

We’ll put the access keys in app.config:

<configuration>
	<appSettings>
		<add key="AmazonAccessKeyId" value="accesskeyid"/>
		<add key="AmazonSecretAccessKey" value="secretaccesskey"/>
	</appSettings>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    </startup>
</configuration>

We’ll insert a simple method to retrieve the necessary AWS credentials as follows:

private static BasicAWSCredentials GetAmazonCredentials()
{
	string secretAccessKey = ConfigurationManager.AppSettings["AmazonSecretAccessKey"];
	string accessKeyId = ConfigurationManager.AppSettings["AmazonAccessKeyId"];
	BasicAWSCredentials basicAwsCredentials = new BasicAWSCredentials(accessKeyId, secretAccessKey);
	return basicAwsCredentials;
}

You’ll need to add a reference to the System.Configuration dll. BasicAWSCredentials is located in the Amazon namespace so you’ll need to reference that as well.

We’ll also need a HTTP client object which will communicate with AWS. This is represented by the AmazonEC2Client object. Now, if you log onto the AWS EC2 management web site then you’ll see that by default the region called US East 1 is selected:

The AmazonEC2Client object has a constructor where you don’t set the selected region in which case the selected region defaults to US East 1 – North Virginia, just like when you log onto the EC2 manager page and the page doesn’t remember what your previous selected region was. This object can be used to send region-independent queries as well, such as finding all available AWS regions, which we’ll look at in a second. Before that we’ll need a method to construct the AmazonEC2Client object:

private static AmazonEC2Client GetAmazonClient(string selectedAmazonRegionEndpoint)
{
	AmazonEC2Client amazonClient = null;
	if (string.IsNullOrEmpty(selectedAmazonRegionEndpoint))
	{
		amazonClient = new AmazonEC2Client(GetAmazonCredentials());
	}
	else
	{
		AmazonEC2Config amazonConfig = new AmazonEC2Config();
		amazonConfig.ServiceURL = "http://" + selectedAmazonRegionEndpoint;
		amazonClient = new AmazonEC2Client(GetAmazonCredentials(), amazonConfig);
	}
	return amazonClient;
}

You see that if you don’t specify a region we return a client specifying our credentials but not the region. Otherwise we provide the region using the AmazonEC2Config object. We’ll see how this section is used in a while, don’t worry about it yet.

Now we have the client object ready so let’s try to find the available regions in Amazon:

private static List<Region> GetAmazonRegions()
{
	AmazonEC2Client amazonEc2Client = GetAmazonClient(null);
	try
	{
		DescribeRegionsRequest describeRegionsRequest = new DescribeRegionsRequest();
		DescribeRegionsResponse describeRegionsResponse = amazonEc2Client.DescribeRegions(describeRegionsRequest);
		DescribeRegionsResult describeRegionsResult = describeRegionsResponse.DescribeRegionsResult;
		List<Region> regions = describeRegionsResult.Region;
		return regions;
	}
	catch
	{
		throw;
	}
}

You’ll see Request and Response objects quite a lot throughout the .NET SDK and this is a good example. We get hold of the list of regions using the request-response pattern. We’ll print the available regions in a separate method:

private static void PrintAmazonRegions(List<Region> regions)
{
	for (int i = 0; i < regions.Count; i++)
	{
		Region region = regions[i];
		Console.WriteLine(string.Format("{0}: Display name: {1}, http endpoint: {2}", i + 1, region.RegionName, region.Endpoint));
	}
}

Let’s connect our methods in Main as follows:

static void Main(string[] args)
{
	List<Region> amazonRegions = GetAmazonRegions();
	PrintAmazonRegions(amazonRegions);
	Console.ReadKey();
}

Run the programme and you should get a list of regions like this:

So far so good! The next step is to find the available AMIs in the selected region. Before we do that let’s alter the existing code so that the user needs to pick a region. Add the following method to read the selected menu point of the user:

private static int GetSelectedRegionOfUser(List<Region> amazonRegions)
{
	Console.Write("Select a region: ");
	string selection = Console.ReadLine();
	int selectableMin = 1;
	int selectableMax = amazonRegions.Count;
	int selectedMenuPoint;
	bool validFormat = int.TryParse(selection, out selectedMenuPoint);
	while (!validFormat || (selectedMenuPoint < selectableMin || selectedMenuPoint > selectableMax))
	{
		Console.WriteLine("Invalid input.");
		Console.Write("Select a region: ");
		selection = Console.ReadLine();
		validFormat = int.TryParse(selection, out selectedMenuPoint);
	}

	return selectedMenuPoint;
}

The revised Main method looks as follows:

static void Main(string[] args)
{
	List<Region> amazonRegions = GetAmazonRegions();
	PrintAmazonRegions(amazonRegions);
	int usersChoice = GetSelectedRegionOfUser(amazonRegions);
        Region selectedRegion = amazonRegions[usersChoice - 1];
	Console.ReadKey();
}

So now we have the selected region. It’s time to look for a suitable AMI in that region. The easiest way to retrieve the list of available machines is the following method:

private static List<Amazon.EC2.Model.Image> GetSuitableImages(Region selectedRegion)
{
	AmazonEC2Client amazonEc2client = GetAmazonClient(selectedRegion.Endpoint);
	try
	{
		DescribeImagesRequest imagesRequest = new DescribeImagesRequest();
		DescribeImagesResponse imagesResponse = amazonEc2client.DescribeImages(imagesRequest);
		DescribeImagesResult imagesResult = imagesResponse.DescribeImagesResult;
		List<Amazon.EC2.Model.Image> images = imagesResult.Image;
		return images;
	}
	catch
	{
		throw;
	}
}

We send in the selected endpoint to the GetAmazonClient method. If you recall then this method will put the selected endpoint into the constructor of the AmazonEC2Client object thereby overriding the default US East 1 region. We then use the request-response objects to retrieve the AMIs from the selected endpoint. However, in its present form the method will return ALL available machines, meaning all public ones and any other that your account may have permission to use. That list is way too long so I recommend that you do not run this method without filtering. You can filter based on the properties of the AMI, e.g. the owner code or the current state. If you are looking for AMIs that belong to a certain owner then you’ll need the code of that owner:

List<String> owners = new List<string>();
owners.Add(ConfigurationManager.AppSettings["AmiSavOwnerId"]);
owners.Add(ConfigurationManager.AppSettings["AmiGclOwnerId"]);
owners.Add(ConfigurationManager.AppSettings["NewAmiOwnerId"]);
imagesRequest.Owner = owners;

As you see it’s possible to fill up a list of strings with the owner IDs which will be assigned to the Owner property of the DescribeImagesRequest object. Go through the available properties of this object to see what other filtering possibilities exist. If you don’t find a ready made property then you can still try the Filter object. Here we’ll filter the AMIs according to their current state:

Filter availabilityFilter = new Filter();
availabilityFilter.Name = "state";
List<String> filterValues = new List<string>();
filterValues.Add("available");
availabilityFilter.Value = filterValues;
List<Filter> filters = new List<Filter>();
filters.Add(availabilityFilter);
imagesRequest.Filter = filters;

It looks a bit cumbersome for a bit of filtering but it goes like this: you define the AMI property by which you want to filter the results in the Name property of the Filter object. Each Filter key can have multiple values hence you need to assign a list of strings. I’m only interested in those AMIs whose ‘state’ property has the value ‘available’. This list of strings will be assigned to the Value property of the Filter object. Then we add this specific filter to the list of filters of the request. So our revised method looks as follows:

private static List<Amazon.EC2.Model.Image> GetSuitableImages(Region selectedRegion)
{
	AmazonEC2Client amazonEc2client = GetAmazonClient(selectedRegion.Endpoint);
	try
	{
		DescribeImagesRequest imagesRequest = new DescribeImagesRequest();

		List<String> owners = new List<string>();
		owners.Add(ConfigurationManager.AppSettings["AmiSavOwnerId"]);
		owners.Add(ConfigurationManager.AppSettings["AmiGclOwnerId"]);
		owners.Add(ConfigurationManager.AppSettings["NewAmiOwnerId"]);
		imagesRequest.Owner = owners;

		Filter availabilityFilter = new Filter();
		availabilityFilter.Name = "state";
		List<String> filterValues = new List<string>();
		filterValues.Add("available");
		availabilityFilter.Value = filterValues;
		List<Filter> filters = new List<Filter>();
		filters.Add(availabilityFilter);
		imagesRequest.Filter = filters;

		DescribeImagesResponse imagesResponse = amazonEc2client.DescribeImages(imagesRequest);
		DescribeImagesResult imagesResult = imagesResponse.DescribeImagesResult;
		List<Amazon.EC2.Model.Image> images = imagesResult.Image;
		return images;
	}
	catch
	{
		throw;
	}
}

The following method will print the images in the Console:

private static void PrintAmis(List<Amazon.EC2.Model.Image> images)
{
	Console.WriteLine(Environment.NewLine);
	Console.WriteLine("Images in the selected region:");
	Console.WriteLine("------------------------------");
	for (int i = 0; i < images.Count; i++)
	{
		Image image = images[i];
		Console.WriteLine(string.Format("{0}: image location: {1}, architecture: {2}", i + 1, image.ImageLocation, image.Architecture));
	}
}

Note that I selected the ImageLocation and Architecture properties of the Image object but feel free to discover all other properties that you can extract from it. The extended Main method looks like this:

static void Main(string[] args)
{
	List<Region> amazonRegions = GetAmazonRegions();
	PrintAmazonRegions(amazonRegions);
	int usersChoice = GetSelectedRegionOfUser(amazonRegions);
	Region selectedRegion = amazonRegions[usersChoice - 1];
	List<Amazon.EC2.Model.Image> imagesInRegion = GetSuitableImages(selectedRegion);
	PrintAmis(imagesInRegion);
	Console.ReadKey();
}

Run the application and if everything goes well then you may see an output similar to the following:

The names of the AMIs will of course be different in your case. We’re now ready to start an image. First let’s get the user’s choice:

private static int GetSelectedRegionOfUser(List<Region> amazonRegions)
{
	Console.Write("Select a region: ");
	string selection = Console.ReadLine();
	int selectableMin = 1;
	int selectableMax = amazonRegions.Count;
	int selectedMenuPoint;
	bool validFormat = int.TryParse(selection, out selectedMenuPoint);
	while (!validFormat || (selectedMenuPoint < selectableMin || selectedMenuPoint > selectableMax))
	{
		Console.WriteLine("Invalid input.");
		Console.Write("Select a region: ");
		selection = Console.ReadLine();
		validFormat = int.TryParse(selection, out selectedMenuPoint);
	}

	return selectedMenuPoint;
}

Main:

static void Main(string[] args)
{
	List<Region> amazonRegions = GetAmazonRegions();
	PrintAmazonRegions(amazonRegions);
	int usersChoice = GetSelectedRegionOfUser(amazonRegions);
	Region selectedRegion = amazonRegions[usersChoice - 1];
	List<Amazon.EC2.Model.Image> imagesInRegion = GetSuitableImages(selectedRegion);
	PrintAmis(imagesInRegion);
	int usersImageChoice = GetSelectedImageOfUser(imagesInRegion);
	Image selectedImage = imagesInRegion[usersImageChoice - 1];
	Console.ReadKey();
}

The code to launch one image instance looks as follows:

private static List<RunningInstance> LaunchImage(Image selectedImage, Region selectedRegion)
{
	AmazonEC2Client amazonEc2client = GetAmazonClient(selectedRegion.Endpoint);
	try
	{
		RunInstancesRequest runInstanceRequest = new RunInstancesRequest();
		runInstanceRequest.ImageId = selectedImage.ImageId;
		runInstanceRequest.InstanceType = "m1.large";
		runInstanceRequest.MinCount = 1;
		runInstanceRequest.MaxCount = 1;
		runInstanceRequest.SecurityGroup = new List<string>() { ConfigurationManager.AppSettings["AmazonSecurityGroupName"] };
		runInstanceRequest.DisableApiTermination = false;

		RunInstancesResponse runInstancesResponse = amazonEc2client.RunInstances(runInstanceRequest);
		RunInstancesResult runInstancesResult = runInstancesResponse.RunInstancesResult;
		Reservation reservation = runInstancesResult.Reservation;
	        List<RunningInstance> runningInstances = reservation.RunningInstance;
		return runningInstances;

	}
	catch
	{
		throw;
	}
}

As before we set the region in the AmazonEC2Client constructor. We then construct the RunInstanceRequest object: we set the selected image ID, the instance type – in this case a large instance -, the number of instances to start – we only want 1 -, and the security group name. We finally determine that we want to be able to terminate the image instance using the API. We then send the request to the AWS API and get a Reservation object back which includes the list of image instances we have started. If only 1 instance was requested then this list will only contain a single element.

Here’s the revised Main method:

static void Main(string[] args)
{
	List<Region> amazonRegions = GetAmazonRegions();
	PrintAmazonRegions(amazonRegions);
	int usersChoice = GetSelectedRegionOfUser(amazonRegions);
	Region selectedRegion = amazonRegions[usersChoice - 1];
	List<Amazon.EC2.Model.Image> imagesInRegion = GetSuitableImages(selectedRegion);
	PrintAmis(imagesInRegion);
	int usersImageChoice = GetSelectedImageOfUser(imagesInRegion);
	Image selectedImage = imagesInRegion[usersImageChoice - 1];
	List<RunningInstance> launchedInstances = LaunchImage(selectedImage, selectedRegion);
	Console.ReadKey();
}

Make your selections in the console. If everything went fine then you’ll see the instance starting up in the AWS console:

In case you don’t see the instance starting up then it may be because you’re not viewing the same region as you selected in the console app. Make sure to select the same region in the AWS manager:

We’re doing good so far. The next step will be to monitor the status of the machine – pending, started etc. and to terminate it. This will be the topic of the next post.