Kubernetes HA Cluster Setup using kubeadm (EC2 + HAProxy)

 Kubernetes HA Cluster Setup using kubeadm (EC2 + HAProxy)

This project demonstrates how to set up a High Availabile Kubernetes Cluster using:-

kubeadm

3 Control Plane Nodes(t2.medium Ubuntu)

HAProxy Load Balancer (t3.micro Ubuntu)

Worker Nodes (t3.micro Ubuntu)

Spin 3 t2.medium EC2 instances IN AWS for Kubernetes Control Plane Nodes:-

==========================================================

Run below commands  on all servers
---------------------------------------------

sudo apt update && sudo apt upgrade -y

Disable swap

--------------------

sudo swapoff -a 

sudo sed -i '/ swap / s/^/#/' /etc/fstab

Enable kernel modules

===================================

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

overlay

br_netfilter

EOF

sudo modprobe overlay 

sudo modprobe br_netfilter

Sysctl settings

===================================

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf 

net.bridge.bridge-nf-call-iptables = 1 

net.ipv4.ip_forward = 1 

net.bridge.bridge-nf-call-ip6tables = 1 

EOF

sudo sysctl --system

Install Container Runtime  - ContainerD

===================================

apt install -y containerd

mkdir -p /etc/containerd

containerd config default > /etc/containerd/config.toml

Enable Systemd cgroup

====================================

sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

sudo systemctl restart containerd 

sudo systemctl enable containerd

Install Kubernetes Components

====================================

sudo apt install -y apt-transport-https ca-certificates curl

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo tee /etc/apt/keyrings/kubernetes-apt-keyring.asc

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.asc] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list

sudo apt update

Install Kubelet, Kubeadm, kubectl

===============================================

sudo apt-get install -y kubelet kubeadm kubectl

sudo apt-mark hold kubelet kubeadm kubectl

Spin Up a t3.micro server in AWS for HAPROXY

HAPROXY Server Setup (Ip Address: 172.31.46.120)

=================

sudo apt install -y haproxy

---Edit config:--------

sudo nano /etc/haproxy/haproxy.cfg

-------Add at bottom:----------

frontend kubernetes bind *:6443 mode tcp option tcplog default_backend k8s-masters

backend k8s-masters mode tcp balance roundrobin option tcp-check server m1 172.31.36.186:6443 check server m2 172.31.42.16:6443 check server m3 172.31.33.101:6443 check

# ==================== Kubernetes API Server Frontend ====================

frontend k8s-api-frontend

    bind *:6443

    mode tcp

    option tcplog

    default_backend k8s-masters

# ==================== Kubernetes Masters Backend ====================

backend k8s-masters

    mode tcp

    balance roundrobin

    

    # Health check - important!

    option tcp-check

    tcp-check connect port 6443

    

    # Server definitions

    server m1 172.31.36.186:6443 check fall 3 rise 2

    server m2 172.31.42.16:6443 check fall 3 rise 2

    server m3 172.31.33.101:6443 check fall 3 rise 2

    

systemctl restart haproxy

=============================================================

 Clean up iptables (important on AWS)

sudo iptables -F

sudo iptables -t nat -F

sudo iptables -t mangle -F

sudo iptables -X

Initialize First Control Plane Node

==============================================================

    

   sudo kubeadm init \

  --control-plane-endpoint "172.31.46.120:6443" \

  --upload-certs \

  --pod-network-cidr=192.168.0.0/16 \

  --node-name $(hostname -s)

    

Setup kubeconfig

=============================================

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

Install CNI (Calico)

=============================================

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Command to regenerate the token if you forget to copy the token:-

=============================================

kubeadm token create --print-join-command


Create the worker Nodes and Join the worker nodes to the control plane with the below command:-

===================================================

Run all the previous Control Plane installation commands and then run the below join command

kubeadm join 172.31.46.120:6443 --token ofefe4.7qwt2h0wbn7jfpg1 --discovery-token-ca-cert-hash sha256:b89d4d35a6ac54d616c1cf7dc26c807bedbacf17dc11ecf4686a2e6ae29868d3

After the installation verify the cluster 
====================================================
kubectl get nodes

 Continous Deployment via ArgoCD

===========================

Create EKS Cluster

===================================================================

eksctl create cluster --name bravo-k8s-argocd --region us-east-1 --nodegroup-name bravo-public-nodes --node-type t3.small  --managed --nodes 2

update .kube/config 

===================================================================

aws eks update-kubeconfig --region us-east-1 --name bravo-k8s-argocd

Verify the config file

===================================================================

kubectl config view

Create ArgoCD NameSpace

===================================================================

kubectl create namespace argocd

Install ArgoCD

===================================================================

kubectl apply -n argocd --server-side --force-conflicts -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Change the ArgoCD service Type to LoadBalancer

===================================================================

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

Initial Admin Password for ArgoCD. You have to change it later

===================================================================

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Login to ArgoCD to complete the CD Part of the application

===================================================================

Create Application--> Name-->Project Name-->Sync Policy(Manual/Automatic)-->Select Source Repo Url-->Revision(Head/Main/Master)-->PATH(Repo Path for example [. / k8s/ etc] ) --> Cluster Url --> namespace -->Click Create

 Harbor Container Registry Setup

 Pre-requisite. Install Docker and Docker Compose

sudo apt update 

sudo apt install -y docker.io 

sudo systemctl enable docker 

sudo systemctl start docker

sudo usermod -aG docker ubuntu 

sudo apt install -y docker-compose 

docker-compose --version

Download the harbor installer:-

wget https://github.com/goharbor/harbor/releases/download/v2.10.0/harbor-offline-installer-v2.10.0.tgz 

tar -xvf harbor-offline-installer-v2.10.0.tgz 

cd harbor

cp harbor.yml.tmpl harbor.yml 

vi harbor.yml

===================
hostname: ipaddress

http: port: 80

Comment HTTPS for insecure lab

https:

port: 443

certificate: /your/cert

private_key: /your/key

===================

/etc/docker/daemon.json

{

"insecure-registries":["ipaddress"]

}

./prepare

./install.sh

Verify by logging in :-

http://23.22.196.203 

Login via shell to push to the new repo:-

docker pull vishnumohan9447/vishnu-tomcat-app:V1

docker login 23.22.196.203 -u admin

docker tag vishnumohan9447/vishnu-tomcat-app:V1  23.22.196.203/prod/vishnu-tomcat-app:V1

docker push 23.22.196.203/prod/vishnu-tomcat-app:V1

 Client VPN Configuration from a Linux Machine to AWS VPC

 // https://docs.aws.amazon.com/pdfs/vpn/latest/clientvpn-admin/client-vpn-admin-guide.pdf

=================================

To generate the server and client certificates and keys and upload them to ACM

1. Clone the OpenVPN easy-rsa repo to your local computer and navigate to the easy-rsa/

easyrsa3 folder.

     $ git clone https://github.com/OpenVPN/easy-rsa.git

     $ cd easy-rsa/easyrsa3

2. Initialize a new PKI environment.

     $ ./easyrsa init-pki

3. To build a new certificate authority (CA), run this command and follow the prompts.

     $ ./easyrsa build-ca nopass

4. Generate the server certificate and key.

     $ ./easyrsa --san=DNS:server build-server-full server nopass

5. Generate the client certificate and key.

Make sure to save the client certificate and the client private key because you will need

them when you configure the client.

     $ ./easyrsa build-client-full client1.domain.tld nopass

You can optionally repeat this step for each client (end user) that requires a client

certificate and key.

6. Copy the server certificate and key and the client certificate and key to a custom folder and

then navigate into the custom folder.Before you copy the certificates and keys, create the custom folder by using the mkdir

command. The following example creates a custom folder in your home directory.

     $ mkdir ~/custom_folder/

     $ cp pki/ca.crt ~/custom_folder/

     $ cp pki/issued/server.crt ~/custom_folder/

     $ cp pki/private/server.key ~/custom_folder/

     $ cp pki/issued/client1.domain.tld.crt ~/custom_folder

     $ cp pki/private/client1.domain.tld.key ~/custom_folder/

     $ cd ~/custom_folder/

7. Upload the server certificate and key and the client certificate and key to ACM. Be sure to

upload them in the same Region in which you intend to create the Client VPN endpoint.

The following commands use the AWS CLI to upload the certificates. To upload the

certificates using the ACM console instead, see Import a certificate in the AWS Certificate

Manager User Guide.

     $ aws acm import-certificate --certificate fileb://server.crt --private-key fileb://server.key --certificate-chain fileb://ca.crt

     $ aws acm import-certificate --certificate fileb://client1.domain.tld.crt --private-key fileb://client1.domain.tld.key --certificate-chain fileb://ca.crt

Then Goto VPC --> Client VPN Endpoints --> Create Client VPN End Point

Enter Name, Description, VPC ID, Select Security Group, Client IpV4 CIDR, Select Server Certificate ARN, Select Use Mutual Authentication, Select Client Certificate ARN, Configure DNS Server

Click Create Client VPN Endpoint 

Goto Target Network Association --> Associate Target Network --> Select any of the subnet that is listed --> Click Associate target Network

Goto Authorization Rule --> Add Authorization Rule --> Enter Destination Network Ipaddress  --> Add Authorization Rule

Download Client Configuration (.ovpn file) and Download and install awsvpn client

Create Profile in the VPN client and select the .ovpn downloaded. Click Connect. You should be able to ping to the private ipaddress of an EC2 instance from your home/office network now