Deploying Java applications with Docker

Docker allows you to create containers from your applications, for more information see: https://www.docker.io/. Docker can be used for many things, one of the options is to replace virtual machines with Docker containers. This article will explain how Docker can be used to setup Tomcat and deploy a Java application.

Why Docker?

Of course there are many tools available to automate the installation of servers and applications. Virtual machines can be used together with tools like Vagrant, Chef, Puppet and many more. But most of them  have their own DSL and some are not for free. That makes it harder to use them as developers and operations are unfamiliar with the tools and DSL’s. Docker has some specific commands, but for the installation of applications like the JDK or Tomcat standard OS commands can be used like apt-get, wget etcetera. Also Docker containers require less resources and startup faster than virtual images as this article will show.

Docker can be used to setup Java application servers for instance test and production machines (although for production it is better to wait until release 1). It is also possible to create a build environment with Jenkins, Nexus, Git, Sonar etc. One container should be used for every application to separate concerns.

The advantages of Docker are quite useful to overcome some of the technical challenges of continuous delivery. Versioning of Docker configuration files is quite easy. Another advantage is that environments can be setup from scratch for every deployment because Docker containers are quickly created and started.

Installing Docker

It is quite easy to install Docker. Instructions can be found on https://docs.docker.io/en/latest/installation. For this article Ubuntu 13.10 was chosen and the commands to install Docker are shown below.

sudo apt-get update
sudo apt-get install linux-image-extra-`uname -r`
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
sudo sh -c "echo deb http://get.docker.io/ubuntu docker main\ > /etc/apt/sources.list.d/docker.list"
sudo apt-get update
sudo apt-get install lxc-docker
sudo docker run -i -t ubuntu /bin/bash

The last step will start a Docker container based on Ubuntu and start bash.

Adding your user to the Docker group

Root access is necessary to work with Docker, but you can also add your user to the docker group. After adding yourself to that group you no longer have to use the ‘sudo’ command.

sudo groupadd docker
sudo gpasswd -a your_username docker
sudo service docker restart

It is necessary to logout and login, restart Ubuntu or enter the command ‘newgrp docker’ before this works correctly.

Create a Docker container with Java and Tomcat

Create a file called ‘Dockerfile’ and add the content below. This is the configuration to install Java and Tomcat on a Ubuntu Saucy (13.10) Docker container that is provided by Docker. There are a lot of official and user supplied container configurations available at https://index.docker.io/. These containers can be used as the basis for your own container.

Dockerfile content:

FROM ubuntu:saucy
# Update Ubuntu
RUN apt-get update && apt-get -y upgrade
# Add oracle java 7 repository
RUN apt-get -y install software-properties-common
RUN add-apt-repository ppa:webupd8team/java
RUN apt-get -y update
# Accept the Oracle Java license
RUN echo "oracle-java7-installer shared/accepted-oracle-license-v1-1 boolean true" | debconf-set-selections
# Install Oracle Java
RUN apt-get -y install oracle-java7-installer
# Install tomcat
RUN apt-get -y install tomcat7
RUN echo "JAVA_HOME=/usr/lib/jvm/java-7-oracle" >> /etc/default/tomcat7
EXPOSE 8080
# Download Slashdot homepage
RUN mkdir /var/lib/tomcat7/webapps/slashdot
RUN wget http://www.slashdot.org -P /var/lib/tomcat7/webapps/slashdot
# Start Tomcat, after starting Tomcat the container will stop. So use a 'trick' to keep it running.
CMD service tomcat7 start && tail -f /var/lib/tomcat7/logs/catalina.out

Build the container using: ‘docker build –t tomcat7 .’. Do not forget to use the ‘.’ at the end. Instead of ‘tomcat7’ you can pick another name if you want. Now start the container using ‘docker run -p 8080:8080 -d tomcat7’. This command makes sure that port 8080 of the container is forwarded to your local port 8080.  After a few seconds the Slashdot page should be available at http://localhost:8080/slashdot/.

Container startup speed

A small test was executed to see how fast the container actually started. This test starts the Docker container and uses ‘wget’ to retrieve a page from within the container.

The commands are placed in a file called ‘dockerSpeed’, the content of the file is shown below:

docker run -p 8080:8080 -d tomcat7
wget localhost:8080/slashdot

The ‘time’ command can be used to see how fast the commands above are executed. Executing ‘time sh dockerSpeed’ gave the following result:

real        0m4.367s
user        0m0.011s
sys         0m0.008s

A few of these tests showed that the execution times were always between 4 and 5 seconds. That’s really quick if you compare that to starting virtual machines.

Container size

Docker creates a container for every command in the Dockerfile. If you have 12 commands in the Dockerfile, then Docker will create 12 containers. This allows the user to select an older container, for instance if you only need a container with Java, you could use that container instead of the one including Tomcat. Luckily only the difference between the containers is stored, which saves quite some diskspace.

Where Docker really wins from virtual machines is if you use multiple containers with the same base. For instance if you have 8 containers each using Ubuntu and Java. You could make a base container with Ubuntu and Java and create containers based on that base image. That way Ubuntu and Java are stored only once in a container, this results in fewer diskspace being used. With virtual machines, you would need that diskspace 8 times as it is not possible to create a common base.

This results in another good point of Docker. Separation of concerns is quite important and can be achieved easily, every application can be deployed in a separate container. So for a buildserver, you could make separate containers for Nexus, Jenkins, Git etcetera.

There are a few options to view the containers. The docker ‘ps’ command shows all active containers. Adding ‘-a’ to the ‘ps’ command also shows the stopped containers. Adding ‘-s’ to the ‘ps’ command shows the size of the containers. The output below from the ‘tomcat7’ container that was created in this article shows that most containers are quite small. Only the containers where bigger things are added like the JDK result in larger containers.

$ docker ps -a -s
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                    NAMES                    SIZE
3d9bd89b5ace        tomcat7:latest      /bin/sh -c service t   10 seconds ago      Up 10 seconds       0.0.0.0:8080->8080/tcp   grave_hawking            51.64 kB
fc116050d899        1ed13b7f9eb1        /bin/sh -c #(nop) CM   15 minutes ago      Exit 0                                       drunk_babbage            0 B
b788f8373ae7        5eb0489aed66        /bin/sh -c wget http   15 minutes ago      Exit 0                                       sleepy_nobel             4.613 kB
f465ff7c6cdf        a8febaa9ed55        /bin/sh -c wget http   15 minutes ago      Exit 0                                       hungry_lumiere           121.3 kB
d4dee2fd7c2f        1938691cd911        /bin/sh -c mkdir /va   15 minutes ago      Exit 0                                       dreamy_euclid            7 B
56f8bff7cfb9        d2e740750084        /bin/sh -c #(nop) EX   15 minutes ago      Exit 0                                       goofy_turing             0 B
1f8639f1841f        52d8cf48f2f3        /bin/sh -c echo "JAV   15 minutes ago      Exit 0                                       grave_bohr               2.074 kB
4cffeada2f59        8811557c9b1b        /bin/sh -c apt-get -   15 minutes ago      Exit 0                                       determined_ptolemy       11 MB
e6a24a05efb3        7d4e8d307140        /bin/sh -c apt-get -   16 minutes ago      Exit 0                                       nostalgic_wozniak        450 MB
090b2652f31e        9f35a5c15127        /bin/sh -c echo "ora   16 minutes ago      Exit 0                                       condescending_brattain   2.764 MB
743d10527376        cce7c0072447        /bin/sh -c apt-get -   16 minutes ago      Exit 0                                       sleepy_ritchie           179 kB
9da7cf4f4ca7        cea346235f02        /bin/sh -c add-apt-r   16 minutes ago      Exit 0                                       jovial_engelbart         863 B
d8c359bdd163        0dcc4e24ae2f        /bin/sh -c apt-get -   17 minutes ago      Exit 0                                       jolly_albattani          33.06 MB
7b184e8b6ece        ubuntu:13.10        /bin/sh -c apt-get u   17 minutes ago      Exit 0                                       trusting_shockley        25.18 MB

Another option is to view the images tree from Docker, the output for the ‘tomcat7’ container is shown below.

$ docker images -tree
└─511136ea3c5a Virtual Size: 0 B
└─1c7f181e78b9 Virtual Size: 0 B
└─9f676bd305a4 Virtual Size: 178 MB Tags: ubuntu:13.10, ubuntu:saucy
└─0dcc4e24ae2f Virtual Size: 215.6 MB
└─cea346235f02 Virtual Size: 249 MB
└─cce7c0072447 Virtual Size: 249 MB
└─9f35a5c15127 Virtual Size: 249.2 MB
└─7d4e8d307140 Virtual Size: 251.9 MB
└─8811557c9b1b Virtual Size: 702.3 MB
└─52d8cf48f2f3 Virtual Size: 713.7 MB
└─d2e740750084 Virtual Size: 713.7 MB
└─1938691cd911 Virtual Size: 713.7 MB
└─a8febaa9ed55 Virtual Size: 713.7 MB
└─5eb0489aed66 Virtual Size: 713.8 MB
└─1ed13b7f9eb1 Virtual Size: 713.8 MB
└─f846774ccd37 Virtual Size: 713.8 MB Tags: tomcat7:latest

Deploy a Java application in Tomcat

The example above used a simple HTML file from Slashdot. Of course that’s not enough, we want to deploy Java applications! That can be done quite easily, just add the following line to the Dockerfile just below the ‘wget’ command for Slashdot.

RUN wget http://tomcat.apache.org/tomcat-7.0-doc/appdev/sample/sample.war -P /var/lib/tomcat7/webapps

Build a Docker container based on the new Dockerfile and run the container. Browsing to http://localhost:8080/sample/ will show the sample Tomcat web application.

Some useful Docker commands

Although Docker basics are fairly simple it still has a lot of options and useful commands.

Testing a bit with Docker will result in quite some containers. Stopping and starting them one by one is not really convenient. The following commands can be used to stop and remove all containers

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

Sometimes you create a container using a Dockerfile but then you realize the container is not working properly. With the command below you can login to the container and see what is going wrong.

docker run -i -t tomcat7 /bin/bash

git - the simple guide

just a simple guide for getting started with git. no deep shit ;)
by Roger Dudler

setup

Download git for OSX
Download git for Windows
Download git for Linux

create a new repository
create a new directory, open it and perform a
git init
to create a new git repository.

checkout a repository
create a working copy of a local repository by running the command
git clone /path/to/repository
when using a remote server, your command will be
git clone username@host:/path/to/repository

workflow

your local repository consists of three "trees" maintained by git. the first one is your Working Directory which holds the actual files. the second one is the Index which acts as a staging area and finally the HEAD which points to the last commit you've made.

add & commit

You can propose changes (add it to the Index) using
git add
git add *
This is the first step in the basic git workflow. To actually commit these changes use
git commit -m "Commit message"
Now the file is committed to the HEAD, but not in your remote repository yet.

pushing changes

Your changes are now in the HEAD of your local working copy. To send those changes to your remote repository, execute
git push origin master
Change master to whatever branch you want to push your changes to.

If you have not cloned an existing repository and want to connect your repository to a remote server, you need to add it with
git remote add origin
Now you are able to push your changes to the selected remote server

branching

Branches are used to develop features isolated from each other. The master branch is the "default" branch when you create a repository. Use other branches for development and merge them back to the master branch upon completion.


create a new branch named "feature_x" and switch to it using
git checkout -b feature_x
switch back to master
git checkout master
and delete the branch again
git branch -d feature_x
a branch is not available to others unless you push the branch to your remote repository
git push origin

update & merge

to update your local repository to the newest commit, execute
git pull
in your working directory to fetch and merge remote changes.
to merge another branch into your active branch (e.g. master), use
git merge
in both cases git tries to auto-merge changes. Unfortunately, this is not always possible and results in conflicts. You are responsible to merge those conflicts manually by editing the files shown by git. After changing, you need to mark them as merged with
git add
before merging changes, you can also preview them by using
git diff

tagging

it's recommended to create tags for software releases. this is a known concept, which also exists in SVN. You can create a new tag named 1.0.0 by executing
git tag 1.0.0 1b2e1d63ff
the 1b2e1d63ff stands for the first 10 characters of the commit id you want to reference with your tag. You can get the commit id by looking at the...

log

in its simplest form, you can study repository history using.. git log
You can add a lot of parameters to make the log look like what you want. To see only the commits of a certain author:
git log --author=bob
To see a very compressed log where each commit is one line:
git log --pretty=oneline
Or maybe you want to see an ASCII art tree of all the branches, decorated with the names of tags and branches:
git log --graph --oneline --decorate --all
See only which files have changed:
git log --name-status
These are just a few of the possible parameters you can use. For more, see git log --help
replace local changes
In case you did something wrong, which for sure never happens ;), you can replace local changes using the command
git checkout --
this replaces the changes in your working tree with the last content in HEAD. Changes already added to the index, as well as new files, will be kept.

If you instead want to drop all your local changes and commits, fetch the latest history from the server and point your local master branch at it like this
git fetch origin
git reset --hard origin/master

useful hints

built-in git GUI
gitk
use colorful git output
git config color.ui true
show log on just one line per commit
git config format.pretty oneline
use interactive adding
git add -i

Getting started with Jenkins, Git and MSBuild

Getting started with Jenkins, Git and MSBuild

In this series:

• Getting started with Jenkins, Git and MSBuild (this post)
• Automatically triggering a Jenkins build on Git commit
• Running NUnit tests with Jenkins

I recently changed jobs and at the new place we are using Jenkins for continuous integration. We are also transitioning to Git from Subversion for source control. I’ve used CruiseControl.Net and Team City before but Jenkins is new to me so I thought it would be interesting to replace my Team City installation at home with Jenkins so I can get a feel for it.

At the time of writing Jenkins is at version 1.602.

1. Install Jenkins on Windows

Jenkins installation is pretty straightforward. Hop over to the Jenkins websiteand downloaded the Windows installation package.

 

The installation package – a Zip archive – contains an MSI installer. Run it. Once that’s completed Jenkins will have been started for you and your web browser will open at http://localhost:8080/, the default for Jenkins.

It’s not immediately obvious but what the MSI has done is installed Jenkins as a Windows service and started it for you.

If you have problems during the installation you can easily start Jenkins manually from the command line. On my system the command looked like this:

?

1	"C:\Program Files (x86)\Java\jre7\bin\java.exe" -jar jenkins.war

You can then go to http://localhost:8080/ to see the Jenkins dashboard. If Jenkins hasn’t installed as a Windows Service you can manually complete that step from the Jenkins management pages. On the Manage Jenkins page you’ll find an ‘Install as Windows Service’ link.

See Install Jenkins as a Windows service.

2. Configure Jenkins to be able to use Git

I had Git installed on my workstation already but if you haven’t you’ll want to do that now.

Once you have Git installed return to the Jenkins dashboard and go to Manage Jenkins > Manage Plugins. Find the Git Plugin on the Available tab (note there’s a filter top-right of the page to help reduce the number of items to look through). Check the tick box next to the Git Plugin and click Install without restart. Note:You may get a message saying to need to restart Jenkins. Just check the restart checkbox and Jenkins will restart for you.

3. Configure Jenkins to be able to use MS Build

If you are building a Visual Studio project you’ll probably want to install theMSBuild Plugin. The process is the same as for installing the Git Plugin. Follow the process in step 2 above to do so.

That’s not all you need to do though. You’ll need to configure Jenkins to know where to find the MSBuild.exe. Go to Manage Jenkins > Configure System.

Scroll down to find the MSBuild section. Here you can add multiple MSBuild entries if you need to to match your requirements (e.g. different .Net versions, 32 or 64 bit, or maybe the version that comes with Visual Studio).

Note: I have been seeing the following warning when adding MSBuild entries:

?

1	C:\Windows\Microsoft.NET\Framework64\v2.0.50727\MSBuild.exe is not a directory on the Jenkins master (but perhaps it exists on some slaves)

I’ve checked my configuration and no slaves are configured. I’ve also had a look at StackOverflow (Jenkins can't find msbuild) but none of the suggested answers work for me. However, MSBuild is being invoked and my builds are working. If I find a resolution I’ll post an update.

4. Setup a build

For my first Jenkins build I chose to use a small project I have that has no external dependencies (no NuGet packages etc.). It’s a very simple configuration service that I use in little projects.

I added a new item and chose Freestyle project before clicking OK.

I was then faced with the configuration page for the project. On this occasion I decided to keep things simple and work against a local Git repository. UnderSource Code Management I chose Git and pointed the Repository Url at my local repository.

The next step was to get MSBuild to build the source. This is done in the Buildsection. When doing this make sure you choose the appropriate instance of MSBuild – choose from the MSBuild instances you configured in step 3 above – and make sure it’s not left as ‘default’. I didn’t bother with any Command Line Arguments but if you need to run specific build configurations (e.g. Release) you can add the appropriate MSBuild arguments in the text box provided.

Once I’d saved the configuration I was ready to go and all I had to do was clickBuild Now to trigger the build.

Once a build has completed the result is visible as a new item in the Build History. If you want to see what happened - maybe you want to start investigating a failed build - you can click on an item in the Build History and from there the Console Output to see the log that was generated by the build.

Wrapping up

So I got my first build to work. First impressions are that Jenkins lacks the slickness of Team City but it appears functional. Next steps for me are to investigate triggering the build when source code is committed and to get my NUnit unit tests to run as part of the build. Updates to follow.

Automatically triggering a Jenkins build on Git commit

In this series:

• Getting started with Jenkins, Git and MSBuild
• Automatically triggering a Jenkins build on Git commit (this post)
• Running NUnit tests with Jenkins

Following on from my previous post, Getting started with Jenkins, Git and MSBuild, the next step is to get Jenkins to build whenever I commit changes to a local Git repository.

Option1 – Polling the repository

Now, it is possible to setup a Build Trigger to get Jenkins to poll for changes in a Git repository. You do this using the Poll SCM option on the project configuration page in Jenkins.

By enabling Poll SCM and entering a cron expression * * * * * in the Scheduletext box we can get Jenkins to poll the repository for changes every minute. You can enter whatever cron expression you like but this serves as a simple example.

Polling the repository is a bit ugly and inefficient. Wouldn’t it be better to get Git to notify Jenkins of a commit and to trigger a build that way? Well you can do just that using ‘Git hooks’.

Option 2 – Triggering a build using Git hooks

Inside every Git repository there’s a hidden .git folder. Inside the .git folder is a hooks subfolder. This folder can hold scripts that are executed by Git when it performs certain operations. This feature is well documented (see Customizing Git - Git Hooks in the Git documentation).

I’m not going to delve into Git hooks here but will just show the steps I took to get Git to trigger a Jenkins build on a Git commit to a local repository by using the post-commit hook.

However, before we get into the details of the post-commit Git hook we need a few other pieces to be in place.

The Git Plugin notifyCommit endpoint

When you install the Git Plugin for Jenkins you get an HTTP endpoint that can be used to trigger Jenkins to check a Git repository for changes and to schedule a build if it finds any. You can test this by hitting the URL directly. Here’s an example that works on my setup (you would need to change the url to match a repository you have access to):

?

1	http://localhost:8080/git/notifyCommit?url=file://c:/Source/Andy.French.Configuration.Service

This feature is described in the ‘Push notification from repository’ section of theGit Plugin documentation. The documentation describes what happens when you hit the endpoint:

“This will scan all the jobs that are configured to check out the specified URL and the optional branches.  If they are also configured with polling, polling will be immediately triggered (and if that finds a change worthy of a build, a build will in turn be triggered). We require the polling configuration on the job so that we only trigger jobs that are supposed to be triggered from changes in the source tree.”

NB: There’s a gotcha here! Make sure you have enabled Poll SCM in your Jenkins build configuration. Don’t specify a schedule, just enable the polling. If you don’t, nothing will happen.

If you see an error like the following check you’ve enabled Poll SCM:

?

1	No Git consumers using SCM API plugin for: file:///c:/Source/Andy.French.Configuration.Service

If you hit the notifyCommit endpoint and there are changes in the repository a build will be triggered. If there are no changes nothing will happen but you can still see that the repository was checked in the Git Polling Log.

Curl

When you installed Git you got curl – a utility for automating HTTP jobs - for free. It’s an exe that lives in the Git bin directory. You can use it to submit requests to an HTTP endpoint like the notifyCommit one we have been looking at and we can do this from a script.

The post-commit Git hook

What we need is for Git to use curl to call the notifyCommit endpoint after a commit takes place. We can do that by using the post-commit Git hook. To get this working add a file called post-commit (no file extension) to the .git\hooksfolder in your repository and open it in a text editor. Add the following to the file:

?

1 2	#!/bin/sh curl http://localhost:8080/git/notifyCommit?url=file://c:/Source/Andy.French.Configuration.Service

You will want to change the URL to one that points to the repository you want to build.

NB: There’s another gotcha here! The first line of the script is essential. This is a bash script and it won’t be executed if that line is missing.

Wrap-up

After putting the post-commit Git hook in place I was able to trigger Jenkins builds whenever I committed changes to the repository. Result! The next step is to run unit tests as part of the build.

Running NUnit tests with Jenkins

In this series:

• Getting started with Jenkins, Git and MSBuild
• Automatically triggering a Jenkins build on Git commit
• Running NUnit tests with Jenkins (this post)

At the time of writing Jenkins is at version 1.602.

Following on from my previous post - Automatically triggering a Jenkins build on Git commit - the next step is to automatically run unit tests as part of the build.

My unit tests are written using NUnit. As with everything in Jenkins getting unit tests to run as part of a build is a fairly manual process which involves adding a new build step to call the NUnit console application directly. Let’s get started.

Step 1 – Install the NUnit binaries

If you haven’t got it already you’ll need to download and install NUnit. You can do so using an MSI or grab the Zip file and extract the binaries to a folder on your machine. Of course you could use Chocolatey to install NUnit.

?

1	chocolatey install nunit

Step 2 - Add a build step to your Jenkins build

Now you have NUnit you’ll need to add a new build step to your Jenkins build. Choose your project from the dashboard.

On the configuration page scroll to the bottom and add click ‘Add build step’. Choose ‘Execute Windows batch command’.

This adds the new build step that you will use to call NUnit. That’s simply a case of calling the NUnit console executable passing in the path to the assembly containing the tests to be run.

Remember to call nunit-console.exe or nunit-console-x86.exe if you’re testing a 32-bit assembly. If you need to run tests against multiple assemblies you can do so by listing them on the command line. NUnit command line documents are here. 

Once you’ve done all that save the changes and click ‘Build now’.

All things being equal you should have a successful build. If you click on the build in the Build History and look at the Console Output you will be able to see the results of the NUnit run.

The output isn’t very slick though, is it? Lets see about getting a nice unit test report working.

Step 3 – Publish NUnit report

First, make sure the NUnit plugin has been installed in Jenkins. If it’s not there you’ll need to add it. Go to Dashboard > Manage Jenkins > Manage Plugins and check your installed plugins. If it’s not there add it. This plugin makes it possible to import NUnit reports from each build into Jenkins so they are displayed with a trend graph and details about which tests that failed.

Next you need to get NUnit to publish its results as an XML file from your build. To do that add ‘/xml=nunit-result.xml’ to the end of the batch command you added in Step 2.

Now add a post-build action to your build definition and select ‘Publish NUnit test result report’.

Enter the name of the XML file you added to the batch command (nunit-result.xml) as the ‘Test report XMLs’ value and save the build configuration.

Run a build and when it completes click on the build ID in the Build History list for the project. You’ll find a new item, Test Result, in the build history. Click on the Test Result link to see the report.

Job done!