Kubernetes HA Cluster Setup using kubeadm (EC2 + HAProxy)

 Kubernetes HA Cluster Setup using kubeadm (EC2 + HAProxy)

This project demonstrates how to set up a High Availabile Kubernetes Cluster using:-

kubeadm

3 Control Plane Nodes(t2.medium Ubuntu)

HAProxy Load Balancer (t3.micro Ubuntu)

Worker Nodes (t3.micro Ubuntu)

Spin 3 t2.medium EC2 instances IN AWS for Kubernetes Control Plane Nodes:-

==========================================================

Run below commands  on all servers
---------------------------------------------

sudo apt update && sudo apt upgrade -y

Disable swap

--------------------

sudo swapoff -a 

sudo sed -i '/ swap / s/^/#/' /etc/fstab

Enable kernel modules

===================================

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

overlay

br_netfilter

EOF

sudo modprobe overlay 

sudo modprobe br_netfilter

Sysctl settings

===================================

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf 

net.bridge.bridge-nf-call-iptables = 1 

net.ipv4.ip_forward = 1 

net.bridge.bridge-nf-call-ip6tables = 1 

EOF

sudo sysctl --system

Install Container Runtime  - ContainerD

===================================

apt install -y containerd

mkdir -p /etc/containerd

containerd config default > /etc/containerd/config.toml

Enable Systemd cgroup

====================================

sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

sudo systemctl restart containerd 

sudo systemctl enable containerd

Install Kubernetes Components

====================================

sudo apt install -y apt-transport-https ca-certificates curl

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo tee /etc/apt/keyrings/kubernetes-apt-keyring.asc

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.asc] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list

sudo apt update

Install Kubelet, Kubeadm, kubectl

===============================================

sudo apt-get install -y kubelet kubeadm kubectl

sudo apt-mark hold kubelet kubeadm kubectl

Spin Up a t3.micro server in AWS for HAPROXY

HAPROXY Server Setup (Ip Address: 172.31.46.120)

=================

sudo apt install -y haproxy

---Edit config:--------

sudo nano /etc/haproxy/haproxy.cfg

-------Add at bottom:----------

frontend kubernetes bind *:6443 mode tcp option tcplog default_backend k8s-masters

backend k8s-masters mode tcp balance roundrobin option tcp-check server m1 172.31.36.186:6443 check server m2 172.31.42.16:6443 check server m3 172.31.33.101:6443 check

# ==================== Kubernetes API Server Frontend ====================

frontend k8s-api-frontend

    bind *:6443

    mode tcp

    option tcplog

    default_backend k8s-masters

# ==================== Kubernetes Masters Backend ====================

backend k8s-masters

    mode tcp

    balance roundrobin

    

    # Health check - important!

    option tcp-check

    tcp-check connect port 6443

    

    # Server definitions

    server m1 172.31.36.186:6443 check fall 3 rise 2

    server m2 172.31.42.16:6443 check fall 3 rise 2

    server m3 172.31.33.101:6443 check fall 3 rise 2

    

systemctl restart haproxy

=============================================================

 Clean up iptables (important on AWS)

sudo iptables -F

sudo iptables -t nat -F

sudo iptables -t mangle -F

sudo iptables -X

Initialize First Control Plane Node

==============================================================

    

   sudo kubeadm init \

  --control-plane-endpoint "172.31.46.120:6443" \

  --upload-certs \

  --pod-network-cidr=192.168.0.0/16 \

  --node-name $(hostname -s)

    

Setup kubeconfig

=============================================

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

Install CNI (Calico)

=============================================

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Command to regenerate the token if you forget to copy the token:-

=============================================

kubeadm token create --print-join-command


Create the worker Nodes and Join the worker nodes to the control plane with the below command:-

===================================================

Run all the previous Control Plane installation commands and then run the below join command

kubeadm join 172.31.46.120:6443 --token ofefe4.7qwt2h0wbn7jfpg1 --discovery-token-ca-cert-hash sha256:b89d4d35a6ac54d616c1cf7dc26c807bedbacf17dc11ecf4686a2e6ae29868d3

After the installation verify the cluster 
====================================================
kubectl get nodes

 Continous Deployment via ArgoCD

===========================

Create EKS Cluster

===================================================================

eksctl create cluster --name bravo-k8s-argocd --region us-east-1 --nodegroup-name bravo-public-nodes --node-type t3.small  --managed --nodes 2

update .kube/config 

===================================================================

aws eks update-kubeconfig --region us-east-1 --name bravo-k8s-argocd

Verify the config file

===================================================================

kubectl config view

Create ArgoCD NameSpace

===================================================================

kubectl create namespace argocd

Install ArgoCD

===================================================================

kubectl apply -n argocd --server-side --force-conflicts -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Change the ArgoCD service Type to LoadBalancer

===================================================================

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

Initial Admin Password for ArgoCD. You have to change it later

===================================================================

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Login to ArgoCD to complete the CD Part of the application

===================================================================

Create Application--> Name-->Project Name-->Sync Policy(Manual/Automatic)-->Select Source Repo Url-->Revision(Head/Main/Master)-->PATH(Repo Path for example [. / k8s/ etc] ) --> Cluster Url --> namespace -->Click Create

 Harbor Container Registry Setup

 Pre-requisite. Install Docker and Docker Compose

sudo apt update 

sudo apt install -y docker.io 

sudo systemctl enable docker 

sudo systemctl start docker

sudo usermod -aG docker ubuntu 

sudo apt install -y docker-compose 

docker-compose --version

Download the harbor installer:-

wget https://github.com/goharbor/harbor/releases/download/v2.10.0/harbor-offline-installer-v2.10.0.tgz 

tar -xvf harbor-offline-installer-v2.10.0.tgz 

cd harbor

cp harbor.yml.tmpl harbor.yml 

vi harbor.yml

===================
hostname: ipaddress

http: port: 80

Comment HTTPS for insecure lab

https:

port: 443

certificate: /your/cert

private_key: /your/key

===================

/etc/docker/daemon.json

{

"insecure-registries":["ipaddress"]

}

./prepare

./install.sh

Verify by logging in :-

http://23.22.196.203 

Login via shell to push to the new repo:-

docker pull vishnumohan9447/vishnu-tomcat-app:V1

docker login 23.22.196.203 -u admin

docker tag vishnumohan9447/vishnu-tomcat-app:V1  23.22.196.203/prod/vishnu-tomcat-app:V1

docker push 23.22.196.203/prod/vishnu-tomcat-app:V1

 Client VPN Configuration from a Linux Machine to AWS VPC

 // https://docs.aws.amazon.com/pdfs/vpn/latest/clientvpn-admin/client-vpn-admin-guide.pdf

=================================

To generate the server and client certificates and keys and upload them to ACM

1. Clone the OpenVPN easy-rsa repo to your local computer and navigate to the easy-rsa/

easyrsa3 folder.

     $ git clone https://github.com/OpenVPN/easy-rsa.git

     $ cd easy-rsa/easyrsa3

2. Initialize a new PKI environment.

     $ ./easyrsa init-pki

3. To build a new certificate authority (CA), run this command and follow the prompts.

     $ ./easyrsa build-ca nopass

4. Generate the server certificate and key.

     $ ./easyrsa --san=DNS:server build-server-full server nopass

5. Generate the client certificate and key.

Make sure to save the client certificate and the client private key because you will need

them when you configure the client.

     $ ./easyrsa build-client-full client1.domain.tld nopass

You can optionally repeat this step for each client (end user) that requires a client

certificate and key.

6. Copy the server certificate and key and the client certificate and key to a custom folder and

then navigate into the custom folder.Before you copy the certificates and keys, create the custom folder by using the mkdir

command. The following example creates a custom folder in your home directory.

     $ mkdir ~/custom_folder/

     $ cp pki/ca.crt ~/custom_folder/

     $ cp pki/issued/server.crt ~/custom_folder/

     $ cp pki/private/server.key ~/custom_folder/

     $ cp pki/issued/client1.domain.tld.crt ~/custom_folder

     $ cp pki/private/client1.domain.tld.key ~/custom_folder/

     $ cd ~/custom_folder/

7. Upload the server certificate and key and the client certificate and key to ACM. Be sure to

upload them in the same Region in which you intend to create the Client VPN endpoint.

The following commands use the AWS CLI to upload the certificates. To upload the

certificates using the ACM console instead, see Import a certificate in the AWS Certificate

Manager User Guide.

     $ aws acm import-certificate --certificate fileb://server.crt --private-key fileb://server.key --certificate-chain fileb://ca.crt

     $ aws acm import-certificate --certificate fileb://client1.domain.tld.crt --private-key fileb://client1.domain.tld.key --certificate-chain fileb://ca.crt

Then Goto VPC --> Client VPN Endpoints --> Create Client VPN End Point

Enter Name, Description, VPC ID, Select Security Group, Client IpV4 CIDR, Select Server Certificate ARN, Select Use Mutual Authentication, Select Client Certificate ARN, Configure DNS Server

Click Create Client VPN Endpoint 

Goto Target Network Association --> Associate Target Network --> Select any of the subnet that is listed --> Click Associate target Network

Goto Authorization Rule --> Add Authorization Rule --> Enter Destination Network Ipaddress  --> Add Authorization Rule

Download Client Configuration (.ovpn file) and Download and install awsvpn client

Create Profile in the VPN client and select the .ovpn downloaded. Click Connect. You should be able to ping to the private ipaddress of an EC2 instance from your home/office network now

KeyCloak Installation and Basic Configuration::

 Installation::

========================

Download keycloak from keycloak.org/downloads

 wget https://github.com/keycloak/keycloak/releases/download/22.0.3/keycloak-22.0.3.tar.gz

 tar -zxvf keycloak-22.0.3.tar.gz

Create Self-signed certificate for Keycloak

openssl req -newkey rsa:2048 -nodes -keyout keycloak-server.key.pem -x509 -days 3650 -out keycloak-server.crt.pem

Copy the key and cert to /usr/share/ssl-cert/

cd keycloak-22.0.3

cd conf

Edit keycloak.conf to  update hostname, certificate and key location in Prod Environment. As this is a test environment, I am using a self-signed certificate and the server local ipaddress.

https-certificate-file=/usr/share/ssl-cert/keycloak-server.crt.pem

https-certificate-key-file=/usr/share/ssl-cert/keycloak-server.key.pem

hostname=172.16.22.136

Goto Keycloak/bin and run the build and start up commands below

./kc.sh build

nohup ./kc.sh start &

Configuration::

==============================

For Keycloak server configuration follow the server administration doc in the Url:- https://www.keycloak.org/docs/latest/server_admin/

Initially login to the Keycloak as admin user. You can create the admin user and password  from the console or set environment variables 

KEYCLOAK_ADMIN=admin
KEYCLOAK_ADMIN_PASSWORD=password

Create New Realm under the Create Realm Menu. A realm manages a set of users, credentials, roles and groups. Master realm is provided as a default realm in Keycloak. Creating multiple realms can enable multiple tenency.

To enable user registration, Goto  under Realm Settings--> Login--> Enable User Registration

To enable Client Authentication, Goto  Clients--> Enable Client Authentication

To Apply new themes, copy the custom theme jar file under the providers folder and run 

"/bin/kc.sh config" command to configure and install the custom providers.

User Management in Keycloak::-

* Self Registration

* From Admin Console

* User Federation

* Automation via API's

Makefile

 If we are compiling a lot of source code files and something goes wrong half way through, it might be nice to be able to pick where we left off in order to finish compiling after we fix the error. Below is an example of a simple Makefile

make command will follow the Makefile and  some of the make command directives are below:-

make clean
make install           
make all
make uninstall

====================

root@debian:~# cat Makefile

all: program

program: main.o  factorial.o

     g++ main.o  factorial.o -o program

main.o: main.cpp

     g++ -c main.cpp

factorial.o: factorial.cpp

     g++ -c factorial.cpp

clean:

     rm -rf *.o program

=====================

=====================

root@debian:~# cat factorial.cpp

#include "functions.h"

int factorial(int n){

   if(n!=1){

      return(n * factorial(n-1));

   } else return 1;

}

======================

======================

root@debian:~# cat functions.h

int factorial(int n);

======================

======================

root@debian:~# cat main.cpp

#include <iostream>

using namespace std;

#include "functions.h"

int main(){

   cout << endl;

   cout << "The factorial of 5 is " << factorial(5) << endl;

   return 0;

}

=======================

Kernel Compilation in Debian from 6.1.0 to 6.5.3

 Kernel Compilation in Debian from 6.1.0 to 6.5.3

uname -r

6.1.0

Download the linux kernel version

wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.3.tar.xz

Untar it

tar -xf linux-6.5.3.tar.xz

Install the necessary dependencies

apt-get install build-essential linux-source bc kmod cpio flex libncurses5-dev libelf-dev libssl-dev dwarves bison

Reboot the server

Run the below commands

make mrproper

This removes any configuration files that might have been accidentally left over from previous builds.

Copy the old .config file

make olddefconfig

Run the below command to make the configuration changes in .config file. 

make menuconfig

Running make localmodconfig will take your current .config and turn off any unused modules.

make localmodconfig

Build the  New Kernel

make -j$(nproc)

Install the kernel modules and the kernel itself:

make modules_install

make install

Reboot the server

shutdown -r now

Run the uname command to know the kernel version

uname -r

6.5.3

Sample OS Patching Ansible Yaml Code

 Sample OS Patching Ansible Yaml Code:

=======================================

---
- name: OS patching of Webservers
  hosts: webservers
  serial: 2
  become: true
  tasks:
    - name : Stop Httpd Service
      service:
        name: httpd
        state: stopped
      when: ansible_distribution == 'CentOS'
    - name : Stop Apache2 Service
      service:
        name: apache2
        state: stopped
      when: ansible_distribution == 'Ubuntu'
    - name : Stop Tomcat Service
      service:
        name: tomcat
        state: stopped
    - name : Stop Keycloak Service
      service:
        name: keycloak
        state: stopped
    - name:  Verify  processes are not running
      shell: if ps -eaf | egrep 'apache|http|tomcat|keycloak'|grep -v grep > /dev/null ;then echo 'process_running';else echo 'process_not_running';fi
      ignore_errors: true
      register: result_process_check
    - name: Run Backup Script prior OS patch
      shell:  sh /opt/scripts/backup_prior_os_patch.sh
    - name: Centos OS paching
      yum:
        name: '*'
        state: latest
      when: result_process_check.stdout == "process_not_running" and ansible_os_family == "RedHat"
    - name: Update Ubuntu repositories cache
      apt:
        update_cache: yes
      when: result_process_check.stdout == "process_not_running" and ansible_os_family == "Debian"
    - name: Update all packages to their latest version
      apt:
        name: "*"
        state: latest
      when: ansible_os_family == "Debian"
    - name: Upgrade the OS (apt-get dist-upgrade)
      apt:
        upgrade: dist
      when: ansible_os_family == "Debian"
    - name: Rebooting the servers
      reboot:
        msg: "Rebooting  Servers After Kernel Patching"
        connect_timeout: 5
        reboot_timeout: 300
        pre_reboot_delay: 0
        post_reboot_delay: 30
        test_command: uptime
      ignore_errors: true

    - name: pause for 180 secs
      pause:
        minutes: 3

Some Usable Adhoc commands

 Some Usable Adhoc commands:

================================

Creating a file on all remote clients

# ansible all –m file –a “path=/home/vishnu/vishnu1 state=touch mode=700”

Deleting a file on all remote clients

# ansible all –m file –a “path=/home/vishnu/vishnu1 state=absent”

Copying a file to remote clients

# ansible all –m copy –a “src=/tmp/vishnu2 dest=/home/vishnu/vishnu2”

Installing package (telnet and httpd-manual)

# ansible all –m yum –a “name=telnet state=present”

# ansible all –m yum –a “name=httpd-manual state=present”. 

Starting httpd package service

# ansible all –m service –a “name=httpd state=started”

Start httpd and enable at boot time

# ansible all –m service –a “name=httpd state=started enabled=yes”

Checking httpd service status on remote client

# ansible all –m shell -a “systemctl status httpd”

Remove httpd package

# ansible all –m yum –a “name=httpd state=absent”

OR

# ansible all –m shell -a “yum remove httpd”.

Creating a user on remote clients

# ansible all –m user –a “name=appu home=/home/appu shell=/bin/bash state=present”

To add a user to a different group

# ansible all –m user –a “name=appu group=vishnu”

Deleting a user on remote clients

# ansible all –m user –a “name=appu home=/home/appu shell=/bin/bash state=absent”

OR

# ansible all –m shell –a “userdel appu”

Getting system information from remote clients

# ansible all –m setup

You can run commands on the remote host without a shell module e.g. reboot client1

# ansible client1 –a “/sbin/reboot”

Mysql server Installation

 

Mysql server Installation:

- name: Install Mysql server, Create database with remote login
  become: yes
  hosts: localhost
  vars:
    Mysql_DB: mysqldb
    Mysql_User: mysql_user
    Mysql_Pass: Password
  tasks:
    - name: Mysql Installation
      package:
        name: "{{item}}"
        state: present
        update_cache: yes
      loop:
        - mysql-server
        - mysql-client
        - python3-mysqldb
        - libmysqlclient-dev
      become: yes

    - name: start and enable mysql service
      service:
        name: mysql
        state: started
        enabled: yes

    - name: create  the user
      mysql_user:
        name: "{{ Mysql_User }}"
        password: "{{ Mysql_Pass }}"
        priv: '*.*:ALL'
        host: '%'
        state: present

    - name: creating the database
      mysql_db:
        name: "{{ Mysql_DB }}"
        state: present

    - name: Enable remote login to mysql
      lineinfile:
         path: /etc/mysql/mysql.conf.d/mysqld.cnf
         regexp: '^bind-address'
         line: 'bind-address = 0.0.0.0'
         backup: yes
      notify:
         - Restart mysql
  handlers:
    - name: Restart mysql
      service:
        name: mysql
        state: restarted

Some Examples of File Module

 Some Examples of File Module:

---
- name: Check if the file or Direcory exists
  hosts: localhost
  become: true
  any_errors_fatal: true
  vars:
    directory: "/tmp"

  tasks:
    - name: Check the status
      stat:
        path: "{{directory}}"
      register: result

    - name: Directory Status
      debug:
        msg: "Directory {{directory}} present"
      when: result.stat.isdir is defined and result.stat.isdir

====================================================================================

---
- name: Check if the file exists
  hosts: localhost
  vars:
    file_path: "/tmp/test"
  become: true
  tasks:
    - name:  Check the file status
      stat:
        path: "{{file_path}}"
      register: result
    - name:  File Exists
      debug:
        msg: "File Exists"
      when: result.stat.exists
    - name: File don't Exists
      debug:
        msg: "File don't exists"
      when: not result.stat.exists

====================================================================================

---
- name: File Ownership
  hosts: localhost
  vars:
    file_name: "/tmp/a"
  become: true
  tasks:
    - name: Change ownership of file
      file:
        path: "{{file_name}}"
        owner: vishnu
        group: vishnu
        mode: 0777


=================================================================================

---
- name: Create Symbolic link
  hosts: localhost
  become: true
  vars:
    sym_link: "/tmp/test1"
    source: "/tmp/test"
  tasks:
    - name: Symbolic link creation
      file:
        src: "{{source}}"
        dest: "{{sym_link}}"
        state: link

==================================================================================


---
- name: Hard Link
  hosts: localhost
  become: true
  vars:
    source: "/tmp/a"
    destin: "/tmp/b"
  tasks:
    - name: "Hard Link"
      file:
        src: "{{source}}"
        dest: "{{destin}}"
        state: hard

=================================================================================

Install Apache on Centos Servers:

 Install Apache on Centos Servers:

---
- name: Install httpd and start the service
  hosts: localhost
  become: true
  tasks:
    - name: install http
      yum:
        name: httpd
        state: latest
    - name: Copy the configuration file
      file:
        src: /tmp/httpd.conf
        dest: /etc/httpd/httpd.conf
      notify:
        Restart Apache
    - name: Start Apache service
      service:
        name: httpd
        state: started
        enabled: true
  handlers:
    - name: Restart Apache
      service:
        name: httpd
        state: restarted

Basic Postgresql Installtion as a Single Node

 Basic Postgresql Installtion as a Single Node:

---
- name: Install postgres
  hosts: localhost
  become: true
  tasks:
    - name: Install postgres
      yum:
        name:
          - postgresql
          - postgresql-server
          - postgresql-contrib
          - postgresql-libs
          - python3-psycopg2
        state: present

    - name: Postgresql initialized or not
      stat:
        path: /var/lib/pgsql/data/pg_hba.conf
      register: result

    - name: InitDB
      shell: postgresql-setup initdb
      when: not result.stat.exists


    - name: Open port for postgresql
      firewalld:
        service: postgresql
        permanent: true
        state: enabled
      notify:
        - Reload firewalld

    - name : Start service
      service: postgresql
      state: started
      enabled: true

  handlers:
    - name: Reload firewalld
      service: firewalld
      state: reloaded