Hardware Configuration

Kubernetes Master & Node configuration
No. of instances	3 - 1 Master, 2 Nodes
AWS Instance type	m3.xlarge
OS	RHEL 7
Storage	200 GB
Mem (GiB)	15GB
vCPU	4

Configuration varies depending on the work load.

Software

Kubernetes version : 1.8.5
Docker version : 1.12.6
Python version : 2.7
Jenkins : 2.60.1
Postgres DB : 9.5.10
Apache Tomcat : 7.0.76

Recommended Network Ports

Kube API Server : 6443
Python service : 5000
Postgres DB : 5432
Jenkins docker container host : 8080
Apache Tomcat : 443

Installation of Kubernetes Cluster

Connect to AWS EC2 instance from Windows using PuTTY

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html

Execute the following commands in the Master and Slave machines

Switch to the root user (sudo -s or sudo su)
yum update
yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional
Note: yum-config-manager --enable is to enable/disable extra and optional repos.
yum update
yum install -y docker
systemctl enable docker && systemctl start docker
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
Execute the following commands to reload and restart docker:

systemctl daemon-reload
systemctl restart docker

Execute the following commands in the Master Machine

# kubeadm init

In case, kubernetes cluster configuration located in $HOME/.kube/config then run the following command

To start using your cluster, run below commands:

# mkdir -p $HOME/.kube

# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

# sudo chown $(id -u):$(id -g) $HOME/.kube/config

# export KUBECONFIG=$HOME/.kube/config

# cp /etc/kubernetes/admin.conf /usr/share/tomcat/ (In RHEL, Tomcat expects the admin.conf to be present in /usr/share/tomcat where BuildOn App is hosted. In tomcat log if it expects kube.config in a different directory place it correspondigly ).

Add “export KUBECONFIG=$HOME/.kube/config” to ~/.bashrc file in order to export the conf file. If this is not added, the export will be applicable only to the current putty session. For a new session, the Kubernetes API will be unreachable.

2. In order to communicate on the network, run the following commands

# kubectl apply -f https://git.io/weave-kube-1.6

3. To create Kubernetes dashboard, run the following commands -

# kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

Dashboard will run as a Kube service either in the Master or in any one of the Slaves. Run “kubectl get svc –all-namespaces” to identify the PORT number of the dashboard. Access the dashboard through https://IP:PORT.

4. Create a cluster role and bind all services then allow anonymous user to access Kube API

If the below kubectl commands are not executed, you will get an error - 'User "system:serviceaccount:default:default" cannot list pods in the namespace "default"'

# kubectl create clusterrolebinding serviceaccounts-cluster-admin --clusterrole=cluster-admin --group=system:serviceaccounts

# kubectl create rolebinding bob-admin-binding --clusterrole=admin --user=system:anonymous --namespace=default

(OR) Use below authentication token (preferred one)

Execute the following command in the Slave machines to connect to the Master machine

After kubeadm init, you will obtain a token as shown below -

# kubeadm join --token <token id> <Primary IP>:6443

Execute it in the slave machines.

2. Post execution, go to the Master machine and run “kubectl get nodes”, and verify if the nodes have joined the cluster.

3. Verify if all pods are up and running.

#kubectl get pods --all-namespaces

You will get an output similar to the one shown below -

BuildOn Framework Setup

BuildOn Scripts

Following are the mandatory files to be located under the directory "/home/ec2-user/BuildOn" to execute the BuildOn.

S.No	Script name	Details
List of Scripts/JARs
1	app.py	Python service (web-hook)
2	buildon.py	Contains the logic to retrieve payload values, insert Buildon commit record to DB, run Kubernetes Pod
3	dbupdate.py	Updates Jenkins live job’s status to 'buildon_reports table'
4	buildon.properties	Contains Kubernetes master IP and framework port

Python modules to be installed in Kube Master

Run the following commands to install Python modules in Kubernetes Master Server.

yum install gcc openssl-devel bzip2-devel
wget https://www.python.org/ftp/python/2.7/Python-2.7.tgz
tar xzf Python-2.7.tgz
Go to the python directory and execute ./configure --enable-optimizations
make altinstall → Prevents from replacing the default python binary file /usr/bin/python
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" → Installs pip utility for the python version

Install Python modules using pip :

pip install wget
pip install docker==2.0.0
pip install pytz
pip install elasticsearch
pip install sys
pip install configparser
pip install gunicorn
pip install gevent
pip install app
pip install flask
pip install pg8000

In case, any other modules are left out while executing app.py, please install them as well.

Execution of Python service

Run the following command from /home/ec2-user/BuildOn folder where app.py located to keep the service always up and running

# nohup gunicorn -k gevent -b 0.0.0.0:5000 -w 2 --timeout 90 app:app > framework.log &

PostgreSQL Setup

PostgreSQL DB usage in BuildOn

PostgreSQL is used to store the extracted data from SCM payload and framework in the DB tables buildon_users, buildon_scmdetails, buildon_preferences, buildon_reports and buildon_servcies. Refer the installation steps to setup PostgreSQL DB for BuildOn.

Installation Steps

Run the following command to setup PostgreSQL

1. yum install postgresql95-server postgresql95-contrib

2. /usr/pgsql-9.5/bin/postgresql95-setup initdb

3. systemctl enable postgresql-9.5.service

4. Change the authentication method ‘ident’ to ‘trust’ in pg_hba.conf file

5. systemctl start postgresql-9.5.service

You can set the postgres user password using following command :

#sudo passwd postgres

How to connect PostgreSQL ?

#sudo -u postgres psql postgres

Check configuaration detail using
# \conninfo

BuildOn Application Setup

BuildOn application is deployed on Apache Tomcat server.

Connect to AWS EC2 instance from Windows using PuTTY

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html

Apache Tomcat as a Service - installation steps

yum install tomcat*
systemctl start tomcat
systemctl enable tomcat [ To start tomcat on boot ]
Add the following lines in /usr/share/tomcat/conf/tomcat-users.xml
<tomcat-users>
<user username="username" password="password" roles="admin-gui,manager-gui"/>
</tomcat-users>

BuildOn application deployment

Deploy the buildon.war using one of the following methods:

Tomcat Manager console
Directly place the buildon.war file in webapps folder(/var/lib/tomcat/webapps)

Post successful deployment of BuildOn, application will be accessible @ https://<KubeMasterIP>/buildon

Enable SSL in tomcat

https://syslint.com/blog/tutorial/how-to-add-ssl-certificate-for-a-domain-in-tomcat-8-server/

Note: Tomcat is advised to be installed as a Service for proper functioning of applications interacting with Git APIs.

BuildOn Application - LDAP Configuration

LDAP configuration in BuildOn

Locate buildon.properties under buildon application resources folder (src/main/resources/buildon.properties)
Set isLDAP=true to use the LDAP for login; isLDAP=false to use DBtable for login, For LDAP set values - ldap.user and ldap.password and isLDAP to true and isopenLDAP to false
ldap.isLDAP=true
LDAP server ip and port
ldap.server=LDAP://<ldap-server-ip>:<ldap-port>
ldap service account username and password to fetch the the user's details
ldap.user=<serviceaccount-username>
ldap.password=<serviceaccount-password>
ldap searchbase attributes
ldap.searchbase=DC=<domain>,DC=<com>
To use openLDAP anonymous user ldap.user and ldap.password set to empty and isLDAP to false
ldap.isopenLDAP=false
Compile and package the code using maven . Deploy the war (buildon.war) to Apache Tomcat Server.

Browser not supported