Introduction to Amazon Code Pipeline with Java part 11: starting with the job agent

Introduction

In the previous post we looked a bit at the web pages that must be up and running for the third party action to work with code pipeline. There are 3 types of URL that must be set up. The external configuration URL defines a web page where the CP user can provide the necessary inputs for the job execution. The execution URL is where the CP user can follow the job execution, i.e. where you can show tables, graphs etc. that show the job progress. Lastly we have the entity URL which is a web page to show some resources or job summary page to the user.

In this post we’ll start looking into the job agent itself. It’s probably the largest component you’ll need to build for the CP integration.

Project type

There are many different project types that will fit a job agent. It must be capable of executing long running processes and HTTPS calls. Also, your language of choice must be compatible with one if the AWS SDKs. There’s a wide variety out there so one of them will probably suit you well. When we at Apica started building the CP integration then CP was still in a preview status and only the Java SDK included the packages for the Code Pipeline library. Therefore we developed in Java which is also the default language of the AWS services. However, if you prefer to build a Python script that runs a process on Linux then you should be fine.

Our job agent is a Maven web project. In its first versions it was a “normal” Java application with no web pages. However, we eventually added a simple web page where people can check the status of the job agent therefore it is now a Maven web project. So you’ll need to be familiar with Java and Maven in order to follow this part of the tutorial.

The POM file

Here’s the complete POM file we currently have in the project. You’ll probably need to fork it some way, e.g. by having different plugins and dependencies, but it should be a good starting point. Make sure you modify the basic repository properties such as the groupId, artifactId and the like:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.apica</groupId>
    <artifactId>AwsCodePipelineBuildRunner</artifactId>
    <version>1.0</version>
    <packaging>war</packaging>

    <name>AwsCodePipelineBuildRunner</name>

    <properties>
        <endorsed.dir>${project.build.directory}/endorsed</endorsed.dir>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
    
    <dependencies>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>2.5.3</version>
        </dependency>

        <dependency>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
            <version>1.2</version>
        </dependency>        
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>LATEST</version>            
        </dependency>
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.0.1</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>joda-time</groupId>
            <artifactId>joda-time</artifactId>
            <version>2.2</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>LATEST</version>
        </dependency>
        <dependency>
            <groupId>org.graylog2</groupId>
            <artifactId>gelfj</artifactId>
            <version>1.1.7</version>
        </dependency>
        <dependency>
            <groupId>org.codehaus.jackson</groupId>
            <artifactId>jackson-mapper-asl</artifactId>
            <version>1.9.13</version>
        </dependency>
        <dependency>
            <groupId>org.codehaus.jackson</groupId>
            <artifactId>jackson-core-asl</artifactId>
            <version>1.9.13</version>
        </dependency>
        <dependency>
            <groupId>commons-daemon</groupId>
            <artifactId>commons-daemon</artifactId>
            <version>1.0.15</version>
        </dependency>
        <dependency>
            <groupId>net.sf.ehcache</groupId>
            <artifactId>ehcache</artifactId>
            <version>2.10.0</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-sts</artifactId>
            <version>1.10.1</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-codepipeline</artifactId>
            <version>LATEST</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-webmvc</artifactId>
            <version>LATEST</version>
        </dependency>
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>servlet-api</artifactId>
            <version>LATEST</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>javax.servlet.jsp</groupId>
            <artifactId>jsp-api</artifactId>
            <version>LATEST</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <compilerArguments>
                        <endorseddirs>${endorsed.dir}</endorseddirs>
                    </compilerArguments>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>2.3</version>
                <configuration>
                    <failOnMissingWebXml>false</failOnMissingWebXml>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.6</version>
                <executions>
                    <execution>
                        <phase>validate</phase>
                        <goals>
                            <goal>copy</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${endorsed.dir}</outputDirectory>
                            <silent>true</silent>
                            <artifactItems>
                                <artifactItem>
                                    <groupId>javax</groupId>
                                    <artifactId>javaee-endorsed-api</artifactId>
                                    <version>7.0</version>
                                    <type>jar</type>
                                </artifactItem>
                            </artifactItems>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

A couple of remarks about the POM:

  • Do not include the entire AWS Java SDK, only the ones required for the job agent, otherwise the deployment package will grow very large with AWS libraries that are never used. The above XML declares the dependency on the S3, STS and Code Pipeline libraries
  • The project is compiled into a .war file for easy deployment on Tomcat. It also fits in well with deployment on Amazon Elastic BeansTalk. We saw some screenshots before about how the project is set up in Beanstalk

Here’s an example of a .war file that the above POM generates in the standard target folder:

War file created in target folder

You can use your favourite Java IDE to build the project. I used NetBeans which is my first choice for any Java development but you can equally go with other Maven-enables IDEs like Eclipse or IntelliJ.

Here’s what the project structure looks like at the time of writing this post:

Job agent project structure in NetBeans

We won’t go into every single detail behind this project as much of it is irrelevant for a general discussion of the job agent. Most of the code is bound to our internal business logic and API settings so that would be an unnecessary distraction. The goal is rather to show enough code so that we end up with a skeleton that you can start customising.

We’ll continue with this in the next post.

View all posts related to Amazon Web Services and Big Data here.

Advertisements

About Andras Nemes
I'm a .NET/Java developer living and working in Stockholm, Sweden.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

ultimatemindsettoday

A great WordPress.com site

Elliot Balynn's Blog

A directory of wonderful thoughts

Robin Sedlaczek's Blog

Developer on Microsoft Technologies

HarsH ReaLiTy

My goal with this blog is to offend everyone in the world at least once with my words… so no one has a reason to have a heightened sense of themselves. We are all ignorant, we are all found wanting, we are all bad people sometimes.

Softwarearchitektur in der Praxis

Wissenswertes zu Webentwicklung, Domain-Driven Design und Microservices

the software architecture

thoughts, ideas, diagrams,enterprise code, design pattern , solution designs

Technology Talks

on Microsoft technologies, Web, Android and others

Software Engineering

Web development

Disparate Opinions

Various tidbits

chsakell's Blog

Anything around ASP.NET MVC,WEB API, WCF, Entity Framework & AngularJS

Cyber Matters

Bite-size insight on Cyber Security for the not too technical.

Guru N Guns's

OneSolution To dOTnET.

Johnny Zraiby

Measuring programming progress by lines of code is like measuring aircraft building progress by weight.

%d bloggers like this: