Introduction to Amazon Code Pipeline with Java part 11: starting with the job agent

Introduction

In the previous post we looked a bit at the web pages that must be up and running for the third party action to work with code pipeline. There are 3 types of URL that must be set up. The external configuration URL defines a web page where the CP user can provide the necessary inputs for the job execution. The execution URL is where the CP user can follow the job execution, i.e. where you can show tables, graphs etc. that show the job progress. Lastly we have the entity URL which is a web page to show some resources or job summary page to the user.

In this post we’ll start looking into the job agent itself. It’s probably the largest component you’ll need to build for the CP integration.

Project type

There are many different project types that will fit a job agent. It must be capable of executing long running processes and HTTPS calls. Also, your language of choice must be compatible with one if the AWS SDKs. There’s a wide variety out there so one of them will probably suit you well. When we at Apica started building the CP integration then CP was still in a preview status and only the Java SDK included the packages for the Code Pipeline library. Therefore we developed in Java which is also the default language of the AWS services. However, if you prefer to build a Python script that runs a process on Linux then you should be fine.

Our job agent is a Maven web project. In its first versions it was a “normal” Java application with no web pages. However, we eventually added a simple web page where people can check the status of the job agent therefore it is now a Maven web project. So you’ll need to be familiar with Java and Maven in order to follow this part of the tutorial.

The POM file

Here’s the complete POM file we currently have in the project. You’ll probably need to fork it some way, e.g. by having different plugins and dependencies, but it should be a good starting point. Make sure you modify the basic repository properties such as the groupId, artifactId and the like:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.apica</groupId>
    <artifactId>AwsCodePipelineBuildRunner</artifactId>
    <version>1.0</version>
    <packaging>war</packaging>

    <name>AwsCodePipelineBuildRunner</name>

    <properties>
        <endorsed.dir>${project.build.directory}/endorsed</endorsed.dir>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
    
    <dependencies>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>2.5.3</version>
        </dependency>

        <dependency>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
            <version>1.2</version>
        </dependency>        
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>LATEST</version>            
        </dependency>
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.0.1</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>joda-time</groupId>
            <artifactId>joda-time</artifactId>
            <version>2.2</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>LATEST</version>
        </dependency>
        <dependency>
            <groupId>org.graylog2</groupId>
            <artifactId>gelfj</artifactId>
            <version>1.1.7</version>
        </dependency>
        <dependency>
            <groupId>org.codehaus.jackson</groupId>
            <artifactId>jackson-mapper-asl</artifactId>
            <version>1.9.13</version>
        </dependency>
        <dependency>
            <groupId>org.codehaus.jackson</groupId>
            <artifactId>jackson-core-asl</artifactId>
            <version>1.9.13</version>
        </dependency>
        <dependency>
            <groupId>commons-daemon</groupId>
            <artifactId>commons-daemon</artifactId>
            <version>1.0.15</version>
        </dependency>
        <dependency>
            <groupId>net.sf.ehcache</groupId>
            <artifactId>ehcache</artifactId>
            <version>2.10.0</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-sts</artifactId>
            <version>1.10.1</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-codepipeline</artifactId>
            <version>LATEST</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-webmvc</artifactId>
            <version>LATEST</version>
        </dependency>
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>servlet-api</artifactId>
            <version>LATEST</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>javax.servlet.jsp</groupId>
            <artifactId>jsp-api</artifactId>
            <version>LATEST</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <compilerArguments>
                        <endorseddirs>${endorsed.dir}</endorseddirs>
                    </compilerArguments>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>2.3</version>
                <configuration>
                    <failOnMissingWebXml>false</failOnMissingWebXml>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.6</version>
                <executions>
                    <execution>
                        <phase>validate</phase>
                        <goals>
                            <goal>copy</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${endorsed.dir}</outputDirectory>
                            <silent>true</silent>
                            <artifactItems>
                                <artifactItem>
                                    <groupId>javax</groupId>
                                    <artifactId>javaee-endorsed-api</artifactId>
                                    <version>7.0</version>
                                    <type>jar</type>
                                </artifactItem>
                            </artifactItems>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

A couple of remarks about the POM:

• Do not include the entire AWS Java SDK, only the ones required for the job agent, otherwise the deployment package will grow very large with AWS libraries that are never used. The above XML declares the dependency on the S3, STS and Code Pipeline libraries
• The project is compiled into a .war file for easy deployment on Tomcat. It also fits in well with deployment on Amazon Elastic BeansTalk. We saw some screenshots before about how the project is set up in Beanstalk

Here’s an example of a .war file that the above POM generates in the standard target folder:

You can use your favourite Java IDE to build the project. I used NetBeans which is my first choice for any Java development but you can equally go with other Maven-enables IDEs like Eclipse or IntelliJ.

Here’s what the project structure looks like at the time of writing this post:

We won’t go into every single detail behind this project as much of it is irrelevant for a general discussion of the job agent. Most of the code is bound to our internal business logic and API settings so that would be an unnecessary distraction. The goal is rather to show enough code so that we end up with a skeleton that you can start customising.

We’ll continue with this in the next post.

View all posts related to Amazon Web Services and Big Data here.