with the growth of pro­gramming pro­jects and the rise of ser­vice ori­en­ted archi­tec­tures, the par­ti­tio­ning of the deve­lo­p­ment pro­cess into more spe­cia­li­zed tasks beco­mes ine­vi­ta­ble. These tasks will usually be sol­ved by dif­fe­rent teams in par­al­lel in order to increase effi­ci­ency, lea­ding to fas­ter pro­gress in the deve­lo­p­ment of the source code. Howe­ver, it crea­tes an addi­tio­nal task of com­bi­ning the pro­gress of the dif­fe­rent teams. Chan­ges intro­du­ced by one team might affect the imple­men­ta­tion of ano­ther, crea­ting con­flicts. While small con­flicts are usually resol­ved quickly, a bela­ted iden­ti­fi­ca­tion might lead to a quick growth and let the deve­lo­pers enter “inte­gra­tion hell”. This term descri­bes the point where mer­ging par­al­lel deve­lo­p­ments of the code and resol­ving asso­cia­ted con­flicts will take up more time than the ori­gi­nal code deve­lo­p­ment itself.

To avoid “inte­gra­tion hell”, con­ti­nuous code inte­gra­tion needs to be done to ensure a timely iden­ti­fi­ca­tion and reso­lu­tion of con­flicts. The­r­e­fore, auto­ma­ted con­ti­nuous inte­gra­tion tools like Jenk­ins have deve­lo­ped to be an indis­pensable tool for soft­ware deve­lo­p­ment teams. They offer the pos­si­bi­lity to auto­ma­ti­cally build and test inte­gra­ted code. This can be done peri­odi­cally or event dri­ven, for exam­ple every time a deve­lo­per pushes new code into a repo­si­tory. The CI tool then reports back the results to the deve­lo­pers which are then able to resolve pos­si­ble con­flicts immediately.

The deve­lo­p­ment of CI soft­ware like Jenk­ins enables to quickly setup a con­ti­nuous inte­gra­tion ser­ver that can be con­fi­gu­red to the spe­ci­fic needs of a pro­ject easily. Howe­ver, hard­ware requi­re­ments for the ser­ver might be dif­fi­cult to esti­mate upfront. Pro­gress in code deve­lo­p­ment is rarely con­stant over time: Upco­ming soft­ware releases will lar­gely increase the amount of code pro­du­ced and the­r­e­fore will increase the demands on the con­ti­nuous inte­gra­tion ser­ver. The asso­cia­ted urgency demand that the ser­ver will still be capa­ble to per­form timely builds and tests of new code. This will lead to hig­her hard­ware requi­re­ments. On the other hand, a pro­ject might hit a period of less deve­lo­p­ment, as deve­lo­pers are con­cer­ned with other projects/problems. If this is the case, idling hard­ware will lead to unneces­sary expenses.

The vary­ing demand makes the CI pipe­line a per­fect tar­get for ela­s­tic com­pu­ting tech­no­lo­gies, as for exam­ple the Kuber­netes con­tai­ner orchestra­tion tool in com­bi­na­tion with cloud-com­pu­ting tech­no­lo­gies. This will enable users to auto­ma­ti­cally spin up ser­vers that build and test newly pro­du­ced code and ter­mi­nate them as soon as they are not nee­ded anymore.

In this short tuto­rial we will create a mini­mal setup of an ela­s­tic CI pipe­line using Jenk­ins within a Kuber­netes clus­ter. This clus­ter can be used locally, or within a sca­lable envi­ron­ment like AWS, Azure or Google cloud ser­vices.  Both Kuber­netes and Jenk­ins offer a large amount of addi­tio­nal setup opti­ons for fur­ther cus­to­miza­tion and can be com­bi­ned with addi­tio­nal tools to com­plete the deve­lo­p­ment pipe­line, for exam­ple code repo­si­to­ries like Git and deploy­ment repo­si­to­ries as for exam­ple Nexus. The fol­lo­wing steps will be per­for­med on AWS-EC2 ins­tances run­ning the Ama­zon Linux 2 AMI. Howe­ver, the steps are adap­ta­ble to other ope­ra­ting sys­tems (but might need a few adjustments).

The Kuber­netes Cluster

Our Kuber­netes clus­ter will con­sist of one mas­ter node and a varia­ble num­ber of worker nodes. As net­work plugin, we will use Flan­nel. We will start by instal­ling the Kuber­netes tools on all nodes. After this pre­pa­ra­tion is done, we will initia­lize the clus­ter on the mas­ter nodes. The worker nodes can then be joi­ned con­se­cu­tively. The fol­lo­wing com­mands should all be run as the root user (not just sudo, as some com­mands will not work) on every node.

sudo su

Assum­ing the ope­ra­ting sys­tem and repo­si­to­ries have been brought up to date via

yum update -y

the setup starts with adding the Kuber­netes repo­si­tory to yum

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

To be able to run the Kuber­netes clus­ter, we need to disable SELi­nux on our node

setenforce 0sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

For yum to accept the Kuber­netes repo­si­tory signa­ture, GnuPG has to be updated

yum install -y http://mirror.centos.org/centos/7/os/x86_64/Packages/gnupg2-2.0.22-5.el7_5.x86_64.rpm

Fur­ther, a few ipta­bles set­tings need to be chan­ged for Flan­nel to work correctly

cat <<EOF >  /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1EOF

sysctl --system

Now, install the Kuber­netes tools

yum install -y docker kubelet kubeadm kubectl --disableexcludes=kubernetes

At this point, it should be che­cked if the br_netfilter is running

lsmod | grep br_netfilter

If this com­mand does not give any out­put, br_netfilter needs to be star­ted with the fol­lo­wing command

modprobe br_netfilter

For the ser­vices nee­ded by Kuber­netes to start auto­ma­ti­cally when the sys­tem boots, the respec­tive ser­vices need to be enab­led in the sys­tem daemon

systemctl enable kubelet.service systemctl enable dockersystemctl start docker

The Mas­ter Node

After run­ning the pre­pa­ra­tion steps of the last sec­tion on every node, it is now time to spin up the clus­ter by running

kubeadm init --pod-network-cidr=10.244.0.0/16

on the node that will serve as the Kuber­netes mas­ter. The com­mand will pro­duce an out­put show­ing how to join addi­tio­nal nodes to the clus­ter. The respec­tive join com­mand can be reco­vered at a later stage by running

kubeadm token create --print-join-command

Note that this com­mand also crea­tes a new token. The tokens expire after 24h and thus need to be crea­ted on a regu­lar basis.

The Kuber­netes clus­ter should be admi­nis­tra­ted by a user dif­fe­rent than root, so we need to exit the root shell

exit

and then run

mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config

to enable the default user to admi­nis­trate our Kuber­netes cluster.

Finally, the Flan­nel net­work plugin needs to be star­ted within the cluster

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml

Note that this com­mand will not be run­nable as root user within the AWS EC2-Ins­tance, as out­bound traf­fic is dis­ab­led for the root user.

Joi­ning Worker Nodes

To join addi­tio­nal EC2 ins­tances as worker nodes, we only need to run the kube­adm join com­mand obtai­ned in the mas­ter node setup on each node. It will look some­thing like this:

kubeadm join 172.31.9.52:6443 --token a9og1w.p9ju926ld39aoagb --discovery-token-ca-cert-hash  sha256:aba42be8add85f3f6bbd8fa76e765a14ab226038b90fc5a967bb3e7e58d716fa

with IP and tokens spe­ci­fic to every setup. The asso­cia­ted IP is the Kuber­netes mas­ter IP and will be nee­ded later. Note that the join com­mand tries to access port 6443 on the Kuber­netes mas­ter node, so that port should be ope­ned for the Kuber­netes worker nodes. We can check if ever­y­thing worked cor­rectly by running

kubectl get nodes

on the Kuber­netes mas­ter node. It should show two nodes (or more if you deci­ded to join addi­tio­nal worker nodes):

Run­ning the Jenk­ins Mas­ter within Kubernetes

Run­ning a web ser­ver like Jenk­ins within Kuber­netes is simple in prin­ci­ple but can get very com­pli­ca­ted if all fea­tures of Kuber­netes should be used. In order to main­tain the sim­pli­city of this tuto­rial, we will rest­rict our­sel­ves to use a dedi­ca­ted host within the Kuber­netes clus­ter to launch the Jenk­ins mas­ter pod. This means we will know the IP address of that spe­ci­fic host in order to com­mu­ni­cate with the Jenk­ins mas­ter. We start by defi­ning a Kuber­netes deploy­ment object on the Kuber­netes mas­ter node by pre­pa­ring a con­fi­gu­ra­tion file jenkins-master-deployment.yaml. Create a file named jenkins-master-deployment.yaml with the fol­lo­wing content:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
        - name: jenkins
          image: schniedersdavid/jenkinsonkubernetes:jenkins
          env:
            - name: JAVA_OPTS
              value: -Djenkins.install.runSetupWizard=false
          ports:
            - name: http-port
              containerPort: 8080
            - name: jnlp-port
              containerPort: 50000
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          emptyDir: {}

app­ly­ing this deploy­ment object using

kubectl apply -f jenkins-master-deployment.yaml

will launch a pod run­ning a jenk­ins con­tai­ner on one of the worker nodes. Using

kubectl describe pods

we can iden­tify which node that is and note down the IP address under which it is reacha­ble. This will be the Jenk­ins mas­ter IP. It will be nee­ded later.

Note that this IP is the AWS private IP. If we want to access this node from out­side the AWS Cloud, we need to extract the public IP of that node from the AWS con­sole. For me, this is 3.16.218.24 (this will be dif­fe­rent for every EC2 instance).

Taking a look at the ports sec­tion of the deploy­ment con­fi­gu­ra­tion file above, we can see that the con­tai­ner will accept con­nec­tions through ports 8080 and 50000. The for­mer is the http port for the Jenk­ins web UI, while the lat­ter will be used for the Java Net­work Laun­ching Pro­to­col (JNLP), which will allow the Jenk­ins mas­ter to run appli­ca­tion on the Jenk­ins slaves later. For these ports to be acces­si­ble within our Kuber­netes clus­ter, we will add two Node­Port ser­vices, ope­ning these spe­ci­fic ports of the Jenk­ins mas­ter pod for appli­ca­tion out­side of the Kuber­netes clus­ter. This will be done by crea­ting two ser­vice con­fi­gu­ra­tion files, jenkins-master-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
  selector:
    app: jenkins

 and jenkins-slave-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins-jnlp
spec:
  type: NodePort
  ports:
    - port: 50000
      targetPort: 50000
  selector:
    app: jenkins

and app­ly­ing them to the cluster

kubectl apply -f jenkins-master-service.yamlkubectl apply -f jenkins-slave-service.yaml

By doing this, we will map the ports 8080 and 50000 within the Jenk­ins mas­ter pod to two ran­domly gene­ra­ted ports bet­ween 30000 and 32767. We can iden­tify these ports by running

kubectl get services

After doing so, we can access the Jenk­ins mas­ter UI using the IP of the node hos­ting the port (we iden­ti­fied it ear­lier using kubectl describe pods) and the Node­Port map­ped to the Jenk­ins html port 8080 (here: 32710). Remem­ber to open that port to your cli­ent IP.

Within this UI, we can now con­fi­gure Jenk­ins as we need it. In this tuto­rial, we will rest­rict our­sel­ves to add the Kuber­netes clus­ter as a cloud to run build jobs within Jenk­ins. To do so, we will visit the “Manage Jenk­ins” tab and navi­gate to “Con­fi­gure Sys­tem”. At the very bot­tom of that page, we see the entry “Cloud”. In the asso­cia­ted drop-down menu, we choose Kuber­netes, which will spawn a large set of new con­fi­gu­ra­tion opti­ons. Here, we will only change a few settings:

• Kuber­netes URL: This is the (private) IP of the Kuber­netes mas­ter (obtainable via the AWS con­sole) on port 6443. For this com­mu­ni­ca­tion to work, the port 6443 needs to be open for other nodes within the cluster.
  
• Jenk­ins URL: This is the (private) IP of the Jenk­ins mas­ter (obtainable from kubectl describe pods, see above) on the Jenk­ins http port (obtainable from kubectl get ser­vices). Here, the port 32710 needs to be open for other nodes within the cluster.
  
• Jenk­ins tun­nel: This is the (private) IP of the Jenk­ins mas­ter (see above) on the Jenk­ins-jnlp port (obtainable from kubectl get ser­vices). Also this port needs to be open for all nodes within the cluster.
  

In addi­tion to that, we  check the box “Disable https cer­ti­fi­cate check”. We fur­ther need to set the cor­rect cre­den­ti­als to allow Jenk­ins to access the Kuber­netes clus­ter. To do so, we click on “add” in the cre­den­ti­als row, we choose “Jenk­ins”, then “Kuber­netes Ser­vice Account” for “kind”. We click on add and then choose the newly added “Secret text” in the cre­den­ti­als drop-down menu. For this ser­vice account to have the cor­rect access rights, we need to run the command

kubectl create clusterrolebinding default-admin --clusterrole cluster-admin --serviceaccount=default:default

on the Kuber­netes mas­ter node. Note that this com­mand will give admin rights to this ser­vice account, which will be a secu­rity issue if your Kuber­netes clus­ter is acces­si­ble by others.

Next, we choose “Add Pod Tem­p­late” at the bot­tom of the page. This will again gene­rate addi­tio­nal fields. Here, we will change “Usage” to “Use this node as much as pos­si­ble”. We will fur­ther define the raw yaml for the Pod as

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: jnlp
    image: jenkins/jnlp-slave:3.27-1-alpine

After saving this con­fi­gu­ra­tion, Jenk­ins is able to start Pods at will in order to run a pro­ject. This can be tes­ted by crea­ting a test pro­ject. On the Jenk­ins main page, we click on “new pro­ject” and choose free­style. We con­fi­gure the pro­ject to run a sin­gle shell command:

cat /dev/zero > /dev/null

This com­mand will run until it is can­ce­led by user interaction.

In order to test the Kuber­netes-Jenk­ins inte­gra­tion, it makes sense to pro­hi­bit builds on the Jenk­ins mas­ter node. We can do this by visi­ting “Manage Jenk­ins -> Con­fi­gure Sys­tem” and set­ting the num­ber of build pro­ces­sors to zero. If we now start a build of our test pro­ject, the Jenk­ins ser­ver should spawn a Jenk­ins slave on any available worker node after some time, which will then run the shell com­mand ente­red before.

What’s next?

With the ela­s­tic Jenk­ins CI pipe­line now run­ning smoothly, sca­ling hard­ware to its cur­rent needs, it is time to inte­grate it into the deve­lo­p­ment pipe­line. A large set of plug­ins for Jenk­ins make it easy to con­nect it to code repo­si­to­ries like Git­Lab, Git­Hub, Bit­bu­cket, and more, allo­wing for cus­to­mi­zed event dri­ven builds. On the other end, plug­ins towards repo­si­to­ries as for exam­ple Sona­type Nexus or Arti­fac­tory allows to auto­ma­ti­cally deploy suc­cessfully built soft­ware. Con­tai­ne­ri­zing these appli­ca­ti­ons as well makes up for a con­sis­tent, easily main­tainable and sca­lable deve­lo­p­ment plat­form. We will dis­cuss these fur­ther steps in an upco­ming blog article, so stay tuned…

We also only scrat­ched the sur­face of Kuber­netes’ poten­tial in this tuto­rial. The Jenk­ins mas­ter is still run­ning on a dedi­ca­ted node that we need to know the IP of to access the web UI. Advan­ced net­wor­king opti­ons within Kuber­netes will help us to address the Jenk­ins mas­ter, regard­less of which node it is run­ning on. Per­sis­tent volume tech­ni­ques will allow to keep the ser­ver con­fi­gu­ra­tion in cases of fail­over, where the Jenk­ins mas­ter pod might be restar­ted on a dif­fe­rent node. These topics pre­sent a more invol­ved deep dive into the com­plex mecha­nisms of Kuber­netes and Docker, which will in large parts be covered in our upco­ming semi­nar series “Ope­ra­tio­na­li­sie­rung von AI-Workloads durch Con­tai­ner-basierte Appli­ka­tio­nen“.