Build Highly-Available PaaS on Redhat Openshift Container Platform using Ansible

Configuration and host preparation

We will install all the openshift packages on ros-master-1 node and install all these packages to rest of the nodes using Ansible playbook

Create SSH Key and copy to all the other nodes for password-less access which will be needed by Ansible . I already have a PEM file which I will use because its already installed on all the other nodes , However you can create a new key using ssh-keygen and copy to all the other nodes using ssh-copy-id command

[centos@ros-master-1 ~]$ ssh-agent $SHELL
[centos@ros-master-1 ~]$ ssh-add cloud.pem
Identity added: cloud.pem (cloud.pem)

Lets verify if we are able to login to other nodes without password

[centos@ros-master-1 ~]$ ssh ros-master-2
Last login: Fri Dec 1 01:40:25 2017 from 172.16.10.27
[centos@ros-master-2 ~]$ exit
logout
Connection to ros-master-2 closed.


[centos@ros-master-1 ~]$ ssh ros-master-3
Last login: Fri Dec 1 01:46:57 2017 from 172.16.10.27
[centos@ros-master-3 ~]$ exit
logout
Connection to ros-master-3 closed.
[centos@ros-master-1 ~]$

Install Openshift Repo, Packages and Dependencies

Rub below commands on ros-master-1

sudo yum install -y centos-release-openshift-origin 

sudo yum install -y origin

sudo yum install -y atomic-openshift-utils

Install Docker on all the openshift nodes using below commands

sudo yum install -y docker

sudo systemctl enable docker

sudo systemctl start docker

Create Ansible host Inventory file

Ansible will use inventory file with openshift specific variables to build openshift cluster and install all the necessary components and software packages on all the nodes. Inventory file can we tweaked with settings you need for your environment. Refer to this Sample Ansible Inventory file which has all the possible options which you can change based on your needs. Below is the Inventory file we will be using.

[OSEv3:children]
nodes
masters
etcd


[OSEv3:vars]
openshift_master_default_subdomain=apps.paas.scaleon.io
ansible_ssh_user=centos
ansible_become=yes
openshift_master_cluster_method=native
openshift_master_cluster_hostname=paas.scaleon.io
openshift_master_cluster_public_hostname=paas.scaleon.io
deployment_type=origin
os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 
'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/htpasswd'}]
openshift_docker_options='--selinux-enabled --insecure-registry 172.30.0.0/16'
openshift_router_selector='region=infra'
openshift_registry_selector='region=infra'
openshift_master_api_port=443
openshift_master_console_port=443
openshift_disable_check=memory_availability,docker_storage
openshift_portal_net=192.18.0.0/16
osm_cluster_network_cidr=10.18.0.0/16
[nodes]
ros-master-1 openshift_schedulable=True ansible_connection=local ansible_become=yes
ros-master-2 openshift_schedulable=True
ros-master-3 openshift_schedulable=True
ros-infra-1 openshift_node_labels="{'region': 'infra'}" openshift_schedulable=True
ros-infra-2 openshift_node_labels="{'region': 'infra'}" openshift_schedulable=True
ros-infra-3 openshift_node_labels="{'region': 'infra'}" openshift_schedulable=True
ros-node-1 openshift_node_labels="{'region': 'node'}" openshift_schedulable=True
ros-node-2 openshift_node_labels="{'region': 'node'}" openshift_schedulable=True
ros-node-3 openshift_node_labels="{'region': 'node'}" openshift_schedulable=True

[masters]
ros-master-1 ansible_connection=local ansible_become=yes
ros-master-2
ros-master-3

[etcd]
ros-master-1 ansible_connection=local ansible_become=yes
ros-master-2
ros-master-3

Note:  I have used openshift_disable_check=memory_availability,docker_storage because my VMs have 4GB RAM each and recommended is 8GB, ansible will fail if RAM is not 8GB.

By default docker uses storage using loopback devices which is not recommended for production use so the check will fail. Either you configure other docker storage driver supported by openshift or in my case I have disabled the check because its a Lab environment.

Note : I am using htpasswd for web authentication which requires us to create user/pass in htpasswd file. Create user and pass with below command on all 3 openshift masters post ansible-playbook completion.

sudo htpasswd -c /etc/origin/htpasswd admin

Also if you like run below command to make admin user as cluster administrator

oc adm policy add-cluster-role-to-user cluster-admin admin

Run the Ansible playbook

Run the ansible playbook from ros-master-1

ansible-playbook -i hosts /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

Ansible playbook will take approx. 10 to 15 mins depending upon the servers and when completed you will see all the PLAY completed with zero failures.

PLAY RECAP ************************************************
localhost : ok=13 changed=0 unreachable=0 failed=0
ros-infra-1 : ok=234 changed=32 unreachable=0 failed=0
ros-infra-2 : ok=234 changed=32 unreachable=0 failed=0
ros-infra-3 : ok=244 changed=56 unreachable=0 failed=0
ros-master-1 : ok=614 changed=81 unreachable=0 failed=0
ros-master-2 : ok=437 changed=51 unreachable=0 failed=0
ros-master-3 : ok=437 changed=51 unreachable=0 failed=0
ros-node-1 : ok=263 changed=33 unreachable=0 failed=0
ros-node-2 : ok=263 changed=33 unreachable=0 failed=0
ros-node-3 : ok=263 changed=33 unreachable=0 failed=0

Verification

[centos@ros-master-1 ~]$ oc version
oc v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://paas.scaleon.io
openshift v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7
[centos@ros-master-1 ~]$

[centos@ros-master-1 ~]$ oc get nodes
NAME STATUS AGE VERSION
ros-infra-1.scaleon.io  Ready 14m v1.6.1+5115d708d7
ros-infra-2.scaleon.io  Ready 14m v1.6.1+5115d708d7
ros-infra-3.scaleon.io  Ready 14m v1.6.1+5115d708d7
ros-master-1.scaleon.io Ready 14m v1.6.1+5115d708d7
ros-master-2.scaleon.io Ready 14m v1.6.1+5115d708d7
ros-master-3.scaleon.io Ready 14m v1.6.1+5115d708d7
ros-node-1.scaleon.io   Ready 14m v1.6.1+5115d708d7
ros-node-2.scaleon.io   Ready 14m v1.6.1+5115d708d7
ros-node-3.scaleon.io   Ready 14m v1.6.1+5115d708d7
[centos@ros-master-1 ~]$


[centos@ros-master-1 ~]$ oc status
In project default on server https://paas.scaleon.io

https://docker-registry-default.apps.paas.scaleon.io (passthrough) (svc/docker-registry)
 dc/docker-registry deploys docker.io/openshift/origin-docker-registry:v3.6.1
 deployment #1 deployed 23 minutes ago - 1 pod

svc/kubernetes - 192.18.0.1 ports 443, 53->8053, 53->8053

https://registry-console-default.apps.paas.scaleon.io (passthrough) (svc/registry-console)
 dc/registry-console deploys docker.io/cockpit/kubernetes:latest
 deployment #1 deployed 21 minutes ago - 1 pod

svc/router - 192.18.32.79 ports 80, 443, 1936
 dc/router deploys docker.io/openshift/origin-haproxy-router:v3.6.1
 deployment #1 deployed 26 minutes ago - 3 pods

View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.
[centos@ros-master-1 ~]$


[centos@ros-master-1 ~]$ oc get all
NAME DOCKER REPO TAGS UPDATED
is/registry-console docker-registry.default.svc:5000/default/registry-console latest 22 minutes ago

NAME REVISION DESIRED CURRENT TRIGGERED BY
dc/docker-registry 1 1 1 config
dc/registry-console 1 1 1 config
dc/router 1 3 3 config

NAME DESIRED CURRENT READY AGE
rc/docker-registry-1 1 1 1 24m
rc/registry-console-1 1 1 1 22m
rc/router-1 3 3 3 26m

NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
routes/docker-registry docker-registry-default.apps.paas.scaleon.io docker-registry <all> passthrough None
routes/registry-console registry-console-default.apps.paas.scaleon.io registry-console <all> passthrough None

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/docker-registry 192.18.208.37 <none> 5000/TCP 24m
svc/kubernetes 192.18.0.1 <none> 443/TCP,53/UDP,53/TCP 15h
svc/registry-console 192.18.175.171 <none> 9000/TCP 22m
svc/router 192.18.32.79 <none> 80/TCP,443/TCP,1936/TCP 26m

NAME READY STATUS RESTARTS AGE
po/docker-registry-1-rrpt9 1/1 Running 0 23m
po/registry-console-1-qhswt 1/1 Running 0 21m
po/router-1-1xn06 1/1 Running 0 25m
po/router-1-bjbr1 1/1 Running 0 25m
po/router-1-p4flt 1/1 Running 0 25m
[centos@ros-master-1 ~]$

Lets verify on web console

Let me know if you run into any issues . I will write another blog on openshift advanced topics like SDN integration and using persistent storage for stateful applications. Thanks