1) Buy a VPS Server
for setting up django web app in centos/Rhel vps system you have to buy vps server from vps provider it will give you server ip address and root password also you will get client web gui for management of server from that you get access to reset server ,reinstall server or boot sever into safe mode and from there you can even change the root password
2) ssh the server
now ssh the sever from any ssh client like putty ,cmd or mobxtrem,etc.
ssh root@192.168.47.101 ---> give root password of server
after that you will get the full access that server
3)Change the root password
once you login to ssh server you should change your root password by passwd commnad
passwd root --> give new password two times
4) add a new user
add a new user so you dont needed to use root for all the times it will make your server more secure
useradd test
passwd test --> give new password two times
now add that test user to wheel group so you can use sudo command by sudo command you can access all system command by that new user
usermod -G wheel test
5) Configure ssh
configure ssh so no one can login to the system via root , edit ssh configuration file and write PermitRootLogin to No
vim /etc/ssh/sshd_config
PermitRootLogin to No
now reboot the system and login via ssh by ssh user@server
6)update the system
now update the system then reboot the system
e.g yum update && reboot
7)Install python3
after sucessfully update install python3
yum install python3
8)install and activate virtual enviroment
after installation of python3 we can use python pip package installer by using pip we can install and create virtual enviroment
by using virtual enviroment we can install python packages virtually that means they only available when we activate the virtual enviroment
pip3 install virtualenv
9)activate virtual enviroment
for activation of virtual enviroment create new directory use source command to activate it
mkdir /djangoenv
virtualenv /djangoenv
source /django/bin/activate
10)install python packages
first create the requirement.txt ,in that file write list of names of python packages
now install it by
pip install -r requirements.txt
11) install sqlite3.8
for django to work in python3 in centos it need the sqlite version 3.8
wget https://kojipkgs.fedoraproject.org//packages/sqlite/3.8.11/1.fc21/x86_64/sqlite-3.8.11-1.fc21.x86_64.rpm
sudo yum install sqlite-3.8.11-1.fc21.x86_64.rpm
12)copy django application to server
copy django application to server by using scp or sftp
scp djangoapplication.zip user@server
13)test the django application
python3 manage.py runserver
14)install apache webserver
install apache webserver
yum install httpd*
15) configure apache server
go to /etc/httpd/conf.d
create web.conf file
type configuration
<virtual host *:80>
servername yoursitename
alias /static /your static directory in django application
alias /media /your media directory in django application
<directory /your static direcory >
require all granted
</directory>
<directory /your media directory >
require all granted
</directory>
<directory /your main app directory where wsgi file located>
<files wsgi.py>
require all granted
</files>
</directory>
WSGIDaemonProcess name_of_your_project python-path = /directory of project
python-home = /directory of enviroment variables'
WSGIProcessGroup name_of_project
WSGIScriptAlias / /your django app directory
</virtualhost>
save and exit
if there is numpy python package in your django app then it will gives time out so add following line to
/etc/httpd/conf/httpd.conf
WSGIApplicationGroup % {GLOBAL}
save and exit
16)Restart apache server
test apache config by
apachectl configtest
restart apache server
systemctl restart httpd
17)go to web browser check your site
Done.
Create Replication Mysql
mysqld --defaults-file=/etc/my.cnf --initialize --user=mysql
In 8.4 Community, the host cache is managed internally.
FLUSH HOST DEPRICATED --> SELECT * FROM performance_schema.host_cache;
CREATE USER 'repl'@'%' IDENTIFIED BY 'password';
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';
FLUSH PRIVILEGES;
CHANGE REPLICATION SOURCE TO
SOURCE_HOST='192.168.241.101',
SOURCE_USER='repl',
SOURCE_PASSWORD='password',
SOURCE_PORT=3360,
SOURCE_AUTO_POSITION=1,
SOURCE_SSL=1,
SOURCE_SSL_CA='/data/ca.pem';
slave my.cnf
[mysqld]
datadir=/data/mysql_server
socket=/var/lib/mysql/mysql.sock
log-error=/data/mysql_server/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
server-id=2
log_bin=mysql-bin
binlog_format=ROW
gtid_mode=ON
enforce_gtid_consistency=ON
get_source_public_key=1
port=3360
user=mysql
symbolic-links=0
# Connection limits (safe for low-memory VM)
max_connections=50
max_user_connections=50
# Packet and temporary table sizes
max_allowed_packet=16M
tmp_table_size=32M
max_heap_table_size=32M
# Sorting and read buffers (per connection, smaller for low RAM)
sort_buffer_size=2M
read_buffer_size=2M
read_rnd_buffer_size=4M
join_buffer_size=2M
# Storage engine
default-storage-engine=InnoDB
key_buffer_size=8M
bulk_insert_buffer_size=8M
# InnoDB settings for small memory
innodb_log_file_size=32M
innodb_print_all_deadlocks=1
innodb_buffer_pool_instances=1
innodb_buffer_pool_size=512M
innodb_read_io_threads=4
innodb_write_io_threads=4
innodb_thread_concurrency=0
innodb_io_capacity=100
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=2
innodb_lock_wait_timeout=50
# Transaction isolation
transaction-isolation=READ-COMMITTED
[root@node2 data]#
============================
master my.cnf
[mysqld]
datadir=/data/mysql_server
socket=/var/lib/mysql/mysql.sock
log-error=/data/mysql_server/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
server-id=1
log_bin=mysql-bin
binlog_format=ROW
gtid_mode=ON
enforce_gtid_consistency=ON
plugin-load-add=mysql_native_password.so
port=3360
user=mysql
symbolic-links=0
# Connection limits (safe for low-memory VM)
max_connections=50
max_user_connections=50
# Packet and temporary table sizes
max_allowed_packet=16M
tmp_table_size=32M
max_heap_table_size=32M
# Sorting and read buffers (per connection, smaller for low RAM)
sort_buffer_size=2M
read_buffer_size=2M
read_rnd_buffer_size=4M
join_buffer_size=2M
# Storage engine
default-storage-engine=InnoDB
key_buffer_size=8M
bulk_insert_buffer_size=8M
# InnoDB settings for small memory
innodb_log_file_size=32M
innodb_print_all_deadlocks=1
innodb_buffer_pool_instances=1
innodb_buffer_pool_size=512M
innodb_read_io_threads=4
innodb_write_io_threads=4
innodb_thread_concurrency=0
innodb_io_capacity=100
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=2
innodb_lock_wait_timeout=50
# Transaction isolation
transaction-isolation=READ-COMMITTED
Secure Monitoring Setup – Prometheus + Node Exporter + Grafana (HTTPS + TLS + Django Embed)
OS: Rocky Linux 8.10 / RHEL compatible
Container runtime: Podman
TLS: Let’s Encrypt or self-signed
Grafana HTTPS port: 42923
Goal:
– node exporter & prometheus not public
– only localhost access
– grafana SSL enabled on random port
– disable anonymous login
– embed inside Django site
– force login authentication
=======================================
Create directories
mkdir -p /opt/node
mkdir -p /opt/prometheus
mkdir -p /opt/grafana/certs
=======================================
2) Copy SSL certificates for Grafana
Use Let’s Encrypt certs (recommended)
cp -L /etc/letsencrypt/live/www.yourdomain.com/fullchain.pem /opt/grafana/certs/fullchain.pem
cp -L /etc/letsencrypt/live/www.yourdomain.com/privkey.pem /opt/grafana/certs/privkey.pem
Fix permissions
chmod 640 /opt/grafana/certs/privkey.pem
chmod 644 /opt/grafana/certs/fullchain.pem
SELinux label
chcon -Rt container_file_t /opt/grafana/certs
=======================================
3) Create Node Exporter HTTPS config file
cat > /opt/node/web.yml <<EOF
tls_server_config:
cert_file: /certs/server.crt
key_file: /certs/server.key
EOF
If using self-signed:
copy cert files
cp /opt/ssl/selfsigned/server.crt /opt/node/
cp /opt/ssl/selfsigned/server.key /opt/node/
=======================================
4) Run Node Exporter (localhost only + HTTPS)
podman run -d
--name node_exporter
-p 127.0.0.1:9100:9100
-v /opt/node/web.yml:/etc/node/web.yml:Z
-v /opt/ssl/selfsigned/server.crt:/certs/server.crt:ro,Z
-v /opt/ssl/selfsigned/server.key:/certs/server.key:ro,Z
quay.io/prometheus/node-exporter
--web.config.file=/etc/node/web.yml
--web.listen-address=127.0.0.1:9100
Test
curl -k https://127.0.0.1:9100/metrics
=======================================
5) Create Prometheus config
cat > /opt/prometheus/prometheus.yml <<EOF
global:
scrape_interval: 5s
scrape_configs:
job_name: 'node'
scheme: https
tls_config:
insecure_skip_verify: true
static_configs:
targets: ['127.0.0.1:9100']
EOF
=======================================
6) Run Prometheus container
podman run -d
--name prometheus
-p 127.0.0.1:9090:9090
-v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:Z
prom/prometheus
Test local access
curl -k https://127.0.0.1:9090/api/v1/targets
You should see "health: up"
=======================================
7) Run Grafana HTTPS on port 42923
Remove any old container first
podman stop grafana || true
podman rm grafana || true
Run new
podman run -d
--name=grafana
--net=host
-v grafana:/var/lib/grafana
-v /opt/grafana/certs/fullchain.pem:/certs/fullchain.pem:ro,Z
-v /opt/grafana/certs/privkey.pem:/certs/privkey.pem:ro,Z
-e GF_SERVER_PROTOCOL=https
-e GF_SERVER_HTTP_PORT=42923
-e GF_SERVER_CERT_FILE=/certs/fullchain.pem
-e GF_SERVER_CERT_KEY=/certs/privkey.pem
-e GF_AUTH_ANONYMOUS_ENABLED=false
-e GF_USERS_ALLOW_SIGN_UP=false
-e GF_SECURITY_ALLOW_EMBEDDING=true
grafana/grafana:latest
Test
curl -k https://127.0.0.1:42923
Expected:
<a href="/login">Found</a>
Browser URL
https://www.yourdomain.com:42923
=======================================
8) First Grafana login
Default user:
admin / admin
Grafana will force you to CHANGE PASSWORD
Create user “xxxx” later in settings.
=======================================
9) Import dashboard ID 1860
In Grafana:
Dashboards → Import
Dashboard ID: 1860
Select Prometheus datasource
=======================================
10) Django embed setup
Create app: serverstats
views.py
from django.contrib.auth.decorators import login_required
from django.shortcuts import render
@login_required
def serverstats_home(request):
return render(request, "serverstats/home.html")
Template home.html
<h2>Server Monitoring Dashboard</h2> <iframe src="https://www.yourdomain.com:42923/d/xxxx?orgId=1&refresh=30s" width="100%" height="900" frameborder="0"> </iframe>
urls.py entry
path("serverstats/", serverstats_home, name="serverstats")
=======================================
11) Security notes
Node exporter is local only ✔
Prometheus is local only ✔
Grafana HTTPS enforced ✔
Random port 42923 ✔
Anonymous Grafana disabled ✔
Django auth required ✔
TLS everywhere ✔
Migrating Web Site From One Vps To Another
update soon
docker basics
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install -y docker-ce docker-ce-cli containerd.io
sudo systemctl enable --now docker
Add user to docker group
sudo usermod -aG docker $USER
Test docker
docker run hello-world
Create file:
/etc/docker/daemon.json
Add configuration:
{
"ipv6": false,
"dns": ["8.8.8.8","1.1.1.1"]
}
Restart docker
systemctl restart docker
Verify
docker run hello-world
docker pull nginx
docker pull ubuntu
docker pull mysql
docker run -d --name mynginx -p 8080:80 nginx
Explanation
-d = run in background
--name mynginx = container name
-p 8080:80 = map container port 80 to host port 8080
docker exec -it mynginx bash
docker stop mynginx
docker start mynginx
docker restart mynginx
docker logs mynginx
docker logs -f mynginx
docker stop mynginx
docker rm mynginx
Remove all stopped containers
docker container prune
docker run -d --name mynginx -p 8081:80 nginx
Container port 80 → Host port 8081
Create configuration file
mkdir -p /data/docker
touch /data/docker/my.cnf
chown 999:999 /data/docker/my.cnf
chmod 644 /data/docker/my.cnf
Run MySQL container
docker run -d \
--name mydb \
-e MYSQL_ROOT_PASSWORD=pass123 \
-v /data/mysql:/var/lib/mysql \
-v /data/docker/my.cnf:/etc/mysql/conf.d/my.cnf \
-p 3306:3306 \
mysql:latest
Create Dockerfile
cat > Dockerfile <<EOF
FROM mysql:8.0
LABEL maintainer="you@example.com"
COPY my.cnf /etc/mysql/conf.d/my.cnf
RUN apt-get update && apt-get install -y \
vim \
net-tools \
&& apt-get clean
EOF
docker images # list images
docker ps # list containers
docker stats # container resource usage
docker logs -f name # view logs
docker inspect name # container config
docker exec -it name bash # enter container
docker top name # processes inside container
docker system df # disk usage
docker run -d \
--name mydb5 \
-e MYSQL_ROOT_PASSWORD=pass123 \
-p 3308:3306 \
--memory="1g" \
--cpus="1.5" \
mysql:8.0
Verify limits
docker inspect mydb5 | grep -i -E "memory|cpus"
Live resource usage
docker stats mydb5
Check inside container
docker exec -it mydb5 cat /sys/fs/cgroup/memory/memory.limit_in_bytes
docker exec -it mydb5 cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
docker exec -it mydb5 cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
docker run -d \
--name mydb10 \
--restart=always \
-e MYSQL_ROOT_PASSWORD=pass123 \
-p 3310:3306 \
mysql:latest
Restart options
no
on-failure
always
unless-stopped
Default network: bridge
Create network
docker network create mynet2
docker network inspect mynet2
Run container on network
docker run -d \
--name mydb12 \
--restart=always \
-e MYSQL_ROOT_PASSWORD=pass123 \
-p 3312:3306 \
--network mynet2 \
mysql:latest
Connect container to multiple networks
docker network connect mynet mydb9
docker network connect mynet1 mydb9
docker network connect mynet2 mydb9
docker network connect mynet3 mydb9
Network types
bridge : containers on same host
host : use host network
none : isolated container
overlay : multi-host cluster networking
Example
docker run -d \
--name web2 \
--restart unless-stopped \
--health-cmd="curl -f http://localhost:80 || exit 1" \
--health-interval=30s \
--health-retries=3 \
--health-timeout=5s \
nginx
Exit codes
0 = success
1 = failure
command missing = failure
docker cp temp-tomcat:/usr/local/tomcat/conf /data/mytomcat
docker run -d \
--name mydb16v8 \
-e MYSQL_ROOT_PASSWORD=pass123 \
-e TZ="Asia/Kolkata" \
--restart unless-stopped \
--health-cmd="mysqladmin ping -h localhost -u root --password=pass123 || exit 1" \
--health-interval=5s \
--health-retries=5 \
-v /data/mysql16v8:/var/lib/mysql \
-v /data/docker/my.cnf:/etc/mysql/conf.d/my.cnf \
-p 3316:3306 \
--memory="1g" \
--cpus="1.5" \
--network mynet2 \
my-mysql-image:8.0
docker run -d \
--name tomcat3 \
--restart unless-stopped \
--health-cmd="curl -f http://localhost:8080 || exit 1" \
--health-interval=30s \
--health-retries=3 \
--health-timeout=5s \
-e TZ="Asia/Kolkata" \
-e JAVA_OPTS="-Xms128m -Xmx256m -Duser.timezone=Asia/Kolkata" \
-p 8083:8080 \
--network mynet2 \
-v /data/tomcat_docker/tomcat_common/warfile/log-api-1.0.war:/usr/local/tomcat/webapps/log-api-1.0.war \
-v /data/tomcat_docker/tomcat3/logs:/usr/local/tomcat/logs \
-v /data/tomcat_docker/tomcat_common/db_properties/db.properties:/usr/local/tomcat/conf/db.properties \
tomcat:9.0.111-jdk8-corretto-al2
docker run -d \
--name nginxlb \
--restart unless-stopped \
--network mynet2 \
-p 80:80 \
-p 443:443 \
-v /data/nginx-lb/nginx.conf:/etc/nginx/nginx.conf:ro \
-v /data/nginx-lb/SSL:/etc/nginx/SSL:ro \
-v /data/nginx-lb/logs:/var/log/nginx \
--health-cmd="sh -c 'echo > /dev/tcp/127.0.0.1/80 || exit 1'" \
--health-interval=30s \
--health-retries=3 \
--health-timeout=5s \
-e TZ="Asia/Kolkata" \
nginx:stable
Test request
curl -k "https://localhost:443/log-api-1.0/log?msgtext=thisistest4011116&status=ok"
Start containers
docker compose up -d
Stop containers
docker compose down
Rebuild containers
docker compose up -d --build
Force recreate
docker compose up -d --force-recreate
==================== OFFLINE KUBERNETES INSTALL (RHEL 8.10) ====================
CLUSTER DETAILS
---------------
MASTER : 192.168.241.160
WORKERS : 192.168.241.161 , 192.168.241.162
K8S : v1.30.14
RUNTIME : containerd
CNI : flannel
ARTIFACTS PATH : /data
packages in /data: conntrack-tools-1.4.4-11.el8.x86_64, containerd.io-1.6.32-3.1.el8.x86_64, cri-tools-1.30.1-150500.1.1.x86_64, ethtool-5.13-2.el8.x86_64, iproute-6.2.0-6.el8_10.x86_64, iproute-tc-6.2.0-6.el8_10.x86_64, iptables-1.8.5-11.el8_9.x86_64, iptables-ebtables-1.8.5-11.el8_9.x86_64, kubeadm-1.30.14-150500.1.1.x86_64, kubectl-1.30.14-150500.1.1.x86_64, kubelet-1.30.14-150500.1.1.x86_64, kubernetes-cni-1.4.0-150500.1.1.x86_64, socat-1.7.4.1-2.el8_10.x86_64, createrepo, bash-auoconnect
================================================================================
STEP 0 : COMMON SETUP (RUN ON ALL NODES)
================================================================================
swapoff -a
sed -i '/swap/d' /etc/fstab
cat <<EOF >/etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat <<EOF >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
EOF
sysctl --system
================================================================================
STEP 1 : CONFIGURE OFFLINE REPO (RUN ON ALL NODES)
================================================================================
cat <<EOF >/etc/yum.repos.d/k8s-offline.repo
[k8s-offline]
name=Kubernetes Offline Repo
baseurl=file:///data/k8s-rpms
enabled=1
gpgcheck=0
EOF
dnf clean all
================================================================================
STEP 2 : INSTALL PACKAGES (RUN ON ALL NODES)
================================================================================
dnf install -y \
containerd.io \
kubeadm kubelet kubectl cri-tools kubernetes-cni \
conntrack-tools iproute iproute-tc iptables iptables-ebtables ethtool socat
systemctl enable --now containerd
systemctl enable kubelet
================================================================================
STEP 3 : CONFIGURE CONTAINERD (RUN ON ALL NODES)
================================================================================
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' \
/etc/containerd/config.toml
systemctl restart containerd
systemctl status containerd
================================================================================
STEP 4 : IMPORT IMAGES (OFFLINE)
================================================================================
# MASTER ONLY
ctr -n k8s.io images import /data/offline/k8s-images.tar
ctr -n k8s.io images import /data/offline/flannel.tar
# WORKERS ONLY
ctr -n k8s.io images import /data/offline/k8s-images.tar
================================================================================
STEP 5 : INITIALIZE CLUSTER (MASTER ONLY)
================================================================================
kubeadm init \
--apiserver-advertise-address=192.168.241.160 \
--pod-network-cidr=10.244.0.0/16
================================================================================
STEP 6 : CONFIGURE kubectl (MASTER ONLY)
================================================================================
mkdir -p $HOME/.kube
cp /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
================================================================================
STEP 7 : INSTALL FLANNEL (MASTER ONLY, OFFLINE)
================================================================================
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl apply -f /data/offline/kube-flannel.yml
kubectl get pods -n kube-system
================================================================================
STEP 8 : JOIN WORKER NODES
================================================================================
# ON MASTER
kubeadm token create --print-join-command
# RUN OUTPUT COMMAND ON EACH WORKER
kubeadm reset -f
rm -rf /etc/cni/net.d
rm -rf /var/lib/cni
rm -rf /var/lib/kubelet/*
systemctl restart containerd
systemctl restart kubelet
kubeadm join 192.168.241.160:6443 \
--token <TOKEN> \
--discovery-token-ca-cert-hash sha256:<HASH>
================================================================================
STEP 9 : VERIFY CLUSTER (MASTER ONLY)
================================================================================
kubectl get nodes -o wide
EXPECTED OUTPUT
---------------
control Ready control-plane
node1 Ready
node2 Ready
==================== OFFLINE KUBERNETES INSTALL COMPLETE =======================
####################### KUBERNETES ETCD FULL LAB (BACKUP + BREAK + RESTORE) #######################
################################ STEP 1: CHECK CLUSTER ############################################
kubectl get nodes
kubectl get pods -A
##############################################################################################
################################ STEP 2: TAKE BACKUP (NO etcdctl ON HOST) ####################
exec -n kube-system etcd-controlnode -- etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --endpoints=https://127.0.0.1:2379 snapshot save /var/lib/etcd-backup3.db
##############################################################################################
################################ STEP 3: BREAK CLUSTER ########################################
kubectl delete deployment webserver-deployment
kubectl delete pod samplepod
kubectl get all
##############################################################################################
################################ STEP 4: STOP ETCD ############################################
mv /etc/kubernetes/manifests/etcd.yaml /tmp/
# Verify
crictl ps | grep etcd
##############################################################################################
################################ STEP 5: RESTORE SNAPSHOT #####################################
ctr -n k8s.io run --rm -t
--mount type=bind,src=/etc/kubernetes/pki/etcd,dst=/etc/kubernetes/pki/etcd,options=rbind:rw
--mount type=bind,src=/var/lib,dst=/var/lib,options=rbind:rw
registry.k8s.io/etcd:3.5.16-0 etcd-restore sh
# Inside container:
ETCDCTL_API=3 etcdctl snapshot restore /var/lib/etcd/etcd-backup.db
--data-dir=/var/lib/etcd-restore
# VERIFY (VERY IMPORTANT)
echo /var/lib/*
exit
##############################################################################################
################################ STEP 6: UPDATE ETCD MANIFEST #################################
vi /etc/kubernetes/manifests/etcd.yaml
# CHANGE THESE THREE PLACES:
--data-dir=/var/lib/etcd-restore
# volumeMounts:
* mountPath: /var/lib/etcd-restore
# volumes:
path: /var/lib/etcd-restore
##############################################################################################
################################ STEP 7: START ETCD ###########################################
mv /tmp/etcd.yaml /etc/kubernetes/manifests/
# Wait 20–30 seconds
##############################################################################################
################################ STEP 8: FIX AUTH (VERY IMPORTANT) ############################
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
sleep 5
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/
mv /etc/kubernetes/manifests/kube-controller-manager.yaml /tmp/
sleep 5
mv /tmp/kube-controller-manager.yaml /etc/kubernetes/manifests/
mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/
sleep 5
mv /tmp/kube-scheduler.yaml /etc/kubernetes/manifests/
##############################################################################################
################################ STEP 9: (OPTIONAL) BYPASS AUTH ###############################
vi /etc/kubernetes/manifests/kube-apiserver.yaml
# Add:
--authorization-mode=AlwaysAllow
##############################################################################################
################################ STEP 10: VERIFY ##############################################
crictl ps
ps -ef | grep etcd
kubectl get nodes
kubectl get pods -A
##############################################################################################
################################ FINAL MEMORY #################################################
Backup → Break → Stop → Restore → Fix Mount → Start → Restart Control Plane → Verify
##############################################################################################
calico-kube-controllers pod was stuck in CrashLoopBackOff and pods could not communicate with the Kubernetes API server via ClusterIP 10.96.0.1:443.
| Component | Detail |
|---|---|
| OS | RHEL 9.6 |
| Kubernetes | v1.29.15 |
| CNI | Calico v3.27.0 |
| Container Runtime | containerd 2.2.2 |
| Node IPs | 192.168.241.140/141/142 |
| Pod CIDR (configured) | 192.168.0.0/16 |
| Service CIDR | 10.96.0.0/12 |
calico-kube-controllers pod stuck in CrashLoopBackOffdial tcp 10.96.0.1:443: i/o timeout10.96.0.110.96.0.1:443 directly from the host (got 403 Forbidden — meaning host-level connectivity was fine)net.ipv4.ip_forward = 1) on all nodes ✅10.96.0.1 existed on all nodes ✅10.96.0.1:443 directly — working on all nodes ✅Host-to-ClusterIP worked but pod-to-ClusterIP timed out. This pointed to a problem specifically with how pod traffic was being NAT'd through the ClusterIP rules.
Running this command on workernode1:
bash
iptables -t nat -L KUBE-SVC-NPX46M4PTMTKRN6Y -v -n
```
Revealed this rule:
```
KUBE-MARK-MASQ tcp -- * * !192.168.0.0/16 10.96.0.1 tcp dpt:443
```
The `!192.168.0.0/16` means — **only masquerade (SNAT) traffic coming from OUTSIDE 192.168.0.0/16**. Traffic from inside that range is excluded from masquerading.
---
## Root Cause
**Pod CIDR `192.168.0.0/16` overlapped with Node IP range `192.168.241.x`.**
This caused a chain reaction:
```
Pod IP: 192.168.212.4
↓
Sends packet to 10.96.0.1:443
↓
kube-proxy KUBE-SERVICES chain matches → forwards to KUBE-SVC-NPX46M4PTMTKRN6Y
↓
KUBE-MARK-MASQ rule checks source IP:
192.168.212.4 is INSIDE 192.168.0.0/16
↓
MASQUERADE is SKIPPED ← problem here
↓
Packet reaches API server (192.168.241.140:6443)
with source IP 192.168.212.4 (pod IP)
↓
API server tries to reply to 192.168.212.4
but has no route back to that pod IP
↓
Connection times out
kube-proxy intentionally excludes pod CIDR from masquerading to avoid unnecessary NAT for pod-to-pod traffic. But when the pod CIDR overlaps with the node network, this optimization breaks pod-to-ClusterIP communication.
| Source | Source IP | In 192.168.0.0/16? | Masqueraded? | Works? |
|---|---|---|---|---|
| Node (host) | 192.168.241.x | Yes | No | ✅ Yes — node IP is routable |
| Pod | 192.168.212.4 | Yes | No | ❌ No — pod IP not directly routable to API server |
Nodes have real routable IPs so replies come back fine even without masquerading. Pods do not — they need SNAT so the reply goes back to the node, which then forwards it to the pod.
Set masqueradeAll: true in kube-proxy configmap:
yaml
iptables:
masqueradeAll: true
This forces SNAT on all pod-to-ClusterIP traffic regardless of source IP, bypassing the overlap problem. This worked but adds NAT overhead on every pod connection.
| Network | Old (broken) | New (correct) |
|---|---|---|
| Pod CIDR | 192.168.0.0/16 |
172.16.0.0/16 |
| Service CIDR | 10.96.0.0/12 |
10.96.0.0/12 |
| Node IPs | 192.168.241.x |
192.168.241.x |
Reinstall command:
bash
kubeadm init \
--pod-network-cidr=172.16.0.0/16 \
--service-cidr=10.96.0.0/12 \
--apiserver-advertise-address=192.168.241.140
With Calico configured to match:
yaml
- name: CALICO_IPV4POOL_CIDR
value: "172.16.0.0/16"
192.168.0.0/16 pod CIDR is just a default, not a requirement — it can and should be changed if your node network uses the same range.Ssl Full Setup (Ca + Server Cert + Verify + Test)
openssl genrsa -out abhilash-ca.key 3072
openssl req -x509 -new -nodes
-key abhilash-ca.key
-sha256 -days 3650
-out abhilash-ca.crt
-subj "/C=IN/ST=Maharashtra/L=Mumbai/O=AbhilashOrg/CN=Abhilash-Root-CA"
openssl genrsa -out server.key 3072
cat < san.cnf
[req]
distinguished_name = dn
req_extensions = req_ext
prompt = no
[dn]
C = IN
ST = Maharashtra
L = Mumbai
O = AbhilashOrg
CN = nginx.local
[req_ext]
subjectAltName = @alt_names
[alt_names]
DNS.1 = nginx.local
DNS.2 = controlnode
IP.1 = 127.0.0.1
IP.2 = 192.168.240.140
EOF
openssl req -new
-key server.key
-out server.csr
-config san.cnf
openssl x509 -req
-in server.csr
-CA abhilash-ca.crt
-CAkey abhilash-ca.key
-CAcreateserial
-out server.crt
-days 825
-sha256
-extensions req_ext
-extfile san.cnf
openssl verify -CAfile abhilash-ca.crt server.crt
openssl x509 -in server.crt -text -noout | grep -A1 "Subject Alternative Name"
openssl x509 -noout -modulus -in server.crt | openssl md5
openssl rsa -noout -modulus -in server.key | openssl md5
openssl s_server -key server.key -cert server.crt -accept 8443
kubectl create secret tls nginx-tls
--cert=server.crt
--key=server.key
🔥 Volume Types Summary
Type Persistent Multi-node Use Case
emptyDir ❌ ❌ temp
hostPath ❌ ❌ node access
PV/PVC ✅ depends DB
NFS ✅ ✅ shared
StorageClass ✅ depends auto
ConfigMap ❌ ❌ config
Secret ❌ ❌ sensitive data
persistentVolumeClaimPolicy
persitentVolumeReclaimPolicy
persistentVolumeReclaimPolicy
=============================
persistenVolumeReclaimPolicy
persistantVolumeReclaimPolicy
pv
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv-secure
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany # NFS supports multiple pods
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys # default auth (explicit)
- nfsvers=4.1 # stable version
- hard # safe retry behavior
nfs:
server: 192.168.1.10 # 👈 your NFS server IP
path: /data/nfsshared/03042026
readOnly: false
Common Units You Should Know
Unit Meaning
Ki Kilobyte
Mi Megabyte
Gi Gigabyte
Ti Terabyte
-------------------------------------------------------------------------
| Mode | Meaning |
| ------------------- | ----------------------- |
| ReadWriteOnce (RWO) | One node only |
| ReadOnlyMany (ROX) | Many pods read only |
| ReadWriteMany (RWX) | Many pods read/write |
==================================================================================
persistentVolumeReclaimPolicy: Retain
| Policy | Behavior |
| -------------------- | --------------- |
| Retain | Keep data |
| Delete | Delete storage |
| Recycle (deprecated) | Clean and reuse |
==============================================================================================
PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-combined
spec:
volumeName: nfs-pv-1 # 👈 exact PV
accessModes:
- ReadWriteMany
storageClassName: "" # 👈 ignore any StorageClass
resources:
requests:
storage: 1Gi
selector:
matchLabels:
app: app1 # 👈 must match PV label
apiVersion: v1
kind: PersistentVolume
metadata:
name: pvc4
labels:
type: pvc4
spec:
volumeName: pv4
accessModes:
- ReadWriteMany
storageClassName: ""
resources:
requests:
storage: 1Gi
selector:
matchLabels:
type: pv4
=========================================================================================================
POD with Volumes
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: sample4
name: sample4-vol-pod
spec:
containers:
- image: nginx:1.25
name: sample4-vol-pod
ports:
- containerPort: 80
command: ["/bin/sh","-c"]
args: ["while true; do echo \"this is test $(date)\" > /data/file.txt; sleep 1; done;"]
env:
- name: TZ
value: IST-5:30
envFrom:
- configMapRef:
name: configmap4
- secretRef:
name: secret4
volumeMounts:
- name: sample4-vol-pod-pvc
mountPath: /app
- name: sample4-vol-pod-emptydir
mountPath: /emptydir
- name: sample4-vol-pod-hostonly
mountPath: /hostpath
- name: configmap4-vol
mountPath: /etc/config
- name: secret5-vol
mountPath: /etc/secret5
volumes:
- name: sample4-vol-pod-pvc
persistentVolumeClaim:
claimName: pvc4
- name: sample4-vol-pod-emptydir
emptyDir: {}
- name: sample4-vol-pod-hostonly
hostPath:
path: /data/k8/hostpath-vol-smaple4
type: DirectoryOrCreate
- name: configmap4-vol
configMap:
name: configmap4
- name: secret5-vol
secret:
secretName: secret5
======================================================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv4
labels:
type: pv4
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: 192.168.241.140
path: /data/nfsshared/date-04march
readOnly: false
====================================================================================
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc4
labels:
type: pvc4
spec:
volumeName: pv4
accessModes:
- ReadWriteMany
storageClassName: ""
resources:
requests:
storage: 1Gi
selector:
matchLabels:
type: pv4
========================================================================================
apiVersion: v1
kind: ConfigMap
metadata:
name: configmap4
labels:
type: configmap4
data:
APP_COLOR: blue
APP_MODE: dev
=======================================================================================
apiVersion: v1
kind: Secret
metadata:
name: secret5
labels:
type: secret5
data:
USERNAME: YWRtaW4K
PASSWORD: YWRtaW4K
=========================================
kubectl create secret generic my-secret \
--from-literal=key=value
| Type | Purpose |
| --------------- | ------------------ |
| generic | Any key-value |
| tls | HTTPS certs |
| docker-registry | Private image pull |
| basic-auth | username/password |
| ssh-auth | SSH keys |
=======================================================================
kubectl create configmap my-config-dir \
--from-file=/path/to/config-dir/
kubectl create configmap my-config \
--from-literal=APP_COLOR=blue \
--from-literal=APP_MODE=dev
===============================
openssl genrsa -out abhilash-ca.key 3072
# 🚀 RabbitMQ Cluster on Kubernetes (Complete Setup + Troubleshooting Guide)
---
# 📌 Objective
Deploy a **3-node RabbitMQ Cluster** on Kubernetes with:
* High Availability
* Persistent Storage (NFS)
* Auto Clustering
* Management UI
* Application connectivity (Tomcat)
---
# 🏗️ Components Created
## 1. Persistent Volumes (NFS)
We created 3 PVs:
* pv-rabbitmq1
* pv-rabbitmq2
* pv-rabbitmq3
Each mapped to:
```text
/data/nfsshared/rabbitmq-pv1
/data/nfsshared/rabbitmq-pv2
/data/nfsshared/rabbitmq-pv3
```
Used:
```yaml
accessModes: ReadWriteOnce
```
👉 Ensures **1 pod = 1 storage**
---
## 2. ConfigMap
Contains:
### enabled_plugins
```erlang
[rabbitmq_management,rabbitmq_peer_discovery_k8s].
```
### rabbitmq.conf
```ini
cluster_formation.peer_discovery_backend = k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = service-rabbitmq-headless
cluster_formation.k8s.hostname_suffix = .service-rabbitmq-headless.default.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
queue_master_locator=min-masters
```
👉 Enables **auto clustering using Kubernetes**
---
## 3. RBAC (CRITICAL)
```yaml
ServiceAccount → rabbitmq
Role → access pods, endpoints
RoleBinding → bind both
```
👉 Required because:
```text
RabbitMQ calls Kubernetes API → needs permission
```
---
## 4. Headless Service
```yaml
name: service-rabbitmq-headless
clusterIP: None
publishNotReadyAddresses: true
```
👉 Enables DNS like:
```text
rabbitmq-0.service-rabbitmq-headless
```
---
## 5. NodePort Service (UI)
```yaml
port: 15672
nodePort: 30072
```
👉 Access UI:
```text
http://<NodeIP>:30072
```
---
## 6. ClusterIP Service (App)
```yaml
name: rabbitmq-svc
port: 5672
```
👉 Used by:
```text
Tomcat → rabbitmq-svc:5672
```
---
## 7. StatefulSet
Key points:
```yaml
serviceName: service-rabbitmq-headless
replicas: 3
```
### ENV:
```yaml
RABBITMQ_DEFAULT_USER=admin
RABBITMQ_DEFAULT_PASS=admin
RABBITMQ_ERLANG_COOKIE=mysecretcookie
RABBITMQ_USE_LONGNAME=true
```
### Volumes:
* PVC → /var/lib/rabbitmq
* ConfigMap → rabbitmq.conf + plugins
👉 Ensures:
* Stable identity
* Persistent data
* Config-driven clustering
---
# ⚙️ FINAL EXECUTION ORDER (VERY IMPORTANT)
👉 Always follow this order:
```bash
kubectl apply -f pv-rabbit-01.yaml
kubectl apply -f pv-rabbit-02.yaml
kubectl apply -f pv-rabbit-03.yaml
kubectl apply -f rbac-rabbitmq.yaml
kubectl apply -f configmap-rabbit.yaml
kubectl apply -f service-rabbitmq-headless.yaml
kubectl apply -f service-rabbitmq-svc.yaml
kubectl apply -f service-rabbitmq-nodeport.yaml
kubectl apply -f StatefulSet-rabbitmq.yaml
```
---
# 🔥 TROUBLESHOOTING JOURNEY
---
## ❌ Issue 1: DNS Not Working
Problem:
```text
rabbitmq-1 not resolving
```
Fix:
```yaml
publishNotReadyAddresses: true
```
---
## ❌ Issue 2: Service Name Mismatch
Problem:
```text
rabbitmq-headless vs service-rabbitmq-headless
```
Fix:
```text
Must match EXACTLY
```
---
## ❌ Issue 3: No rabbitmq.conf
Fix:
Added clustering config
---
## ❌ Issue 4: 403 Error (CRITICAL)
Log:
```text
Failed to fetch nodes from Kubernetes API: 403
```
Fix:
Added RBAC
---
## ❌ Issue 5: Short vs Long Names
Error:
```text
epmd nxdomain
```
Fix:
```yaml
RABBITMQ_USE_LONGNAME=true
```
---
## ❌ Issue 6: Cluster Join Failure
Error:
```text
tables_not_present
mnesia_not_running
```
👉 Root cause:
```text
Pods not ready at same time (timing issue)
```
---
## ❌ Issue 7: Cluster Not Forming
Final log:
```text
Starting as a blank standalone node
```
👉 Reason:
```text
Retry failed → node becomes standalone
```
---
# 🧠 WHY THIS HAPPENS
RabbitMQ:
```text
Cluster formation happens ONLY at startup
```
If peers not ready → join fails
---
# 🔧 FINAL FIXES APPLIED
* Enabled RBAC ✅
* Enabled longnames ✅
* Fixed serviceName ✅
* Fixed DNS ✅
* Added retry logic ✅
* Restarted pods cleanly ✅
---
# 📊 FINAL RESULT
```bash
rabbitmqctl cluster_status
```
Output:
```text
rabbit@rabbitmq-0
rabbit@rabbitmq-1
rabbit@rabbitmq-2
```
---
# 🎯 WHAT WE ACHIEVED
✅ 3-node RabbitMQ cluster
✅ Auto discovery via Kubernetes
✅ Persistent storage
✅ UI access
✅ App connectivity
✅ HA-ready setup
---
# ⚠️ ALTERNATIVES
| Approach | Result |
| -------------- | ------------------------ |
| No RBAC | No clustering ❌ |
| Manual join | Works but not stable ⚠️ |
| Classic config | Static, not scalable ❌ |
| Helm chart | Best production option ✅ |
---
# 🧠 FINAL LEARNING
* Kubernetes = dynamic → needs API
* RabbitMQ = startup-based clustering
* RBAC = mandatory
* Headless service = must
* Longnames = required
* Timing = critical
---
# 🚀 NEXT STEPS
* Create quorum queues
* Test failover (kill pod)
* Connect Tomcat producer/consumer
* Monitor cluster
---
# 📌 FINAL CONCLUSION
You successfully built a **production-grade RabbitMQ cluster on Kubernetes**
and solved real-world issues like:
* DNS
* RBAC
* Clustering
* Node naming
* Startup timing
---
==================================configmap-rabbit.yaml========================================
apiVersion: v1
kind: ConfigMap
metadata:
name: configmap-rabbit
labels:
type: configmap-rabbit
data:
enabled_plugins: |
[rabbitmq_management,rabbitmq_peer_discovery_k8s].
rabbitmq.conf: |
cluster_formation.peer_discovery_backend = k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = service-rabbitmq-headless
cluster_formation.k8s.hostname_suffix = .service-rabbitmq-headless.default.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
queue_master_locator=min-masters
==================================pv-rabbit-01.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq1
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv1
readOnly: false
==================================pv-rabbit-02.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq2
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv2
readOnly: false
==================================pv-rabbit-03.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq3
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv3
readOnly: false
==================================rbac-rabbitmq.yaml========================================
apiVersion: v1
kind: ServiceAccount
metadata:
name: rabbitmq
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rabbitmq
namespace: default
rules:
- apiGroups: [""]
resources:
- endpoints
- pods
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rabbitmq
namespace: default
subjects:
- kind: ServiceAccount
name: rabbitmq
namespace: default
roleRef:
kind: Role
name: rabbitmq
apiGroup: rbac.authorization.k8s.io
==================================service-rabbitmq-headless.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: service-rabbitmq-headless
labels:
type: service-rabbitmq-headless
spec:
clusterIP: None
publishNotReadyAddresses: true
selector:
app: rabbitmq
ports:
- name: amqp
port: 5672
- name: management
port: 15672
- name: epmd
port: 4369
- name: cluster-rpc
port: 25672
==================================service-rabbitmq-nodeport.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-nodeport
labels:
type: rabbitmq-nodeport
spec:
type: NodePort
selector:
app: rabbitmq
ports:
- name: management
port: 15672
targetPort: 15672
nodePort: 30072
==================================service-rabbitmq-svc.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-svc
labels:
type: rabbitmq-svc
spec:
type: ClusterIP
selector:
app: rabbitmq
ports:
- name: amqp
port: 5672
targetPort: 5672
==================================StatefulSet-rabbitmq.yaml========================================
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rabbitmq
labels:
type: rabbitmq
spec:
serviceName: service-rabbitmq-headless
replicas: 3
selector:
matchLabels:
app: rabbitmq
template:
metadata:
labels:
app: rabbitmq
spec:
serviceAccountName: rabbitmq
containers:
- name: rabbitmq
image: rabbitmq:3.12-management
ports:
- containerPort: 5672
- containerPort: 15672
env:
- name: RABBITMQ_DEFAULT_USER
value: "admin"
- name: RABBITMQ_DEFAULT_PASS
value: "admin"
- name: RABBITMQ_ERLANG_COOKIE
value: "mysecretcookie"
- name: RABBITMQ_USE_LONGNAME
value: "true"
volumeMounts:
- name: data
mountPath: /var/lib/rabbitmq
- name: config
mountPath: /etc/rabbitmq/enabled_plugins
subPath: enabled_plugins
- name: rabbitconf
mountPath: /etc/rabbitmq/rabbitmq.conf
subPath: rabbitmq.conf
volumes:
- name: config
configMap:
name: configmap-rabbit
- name: rabbitconf
configMap:
name: configmap-rabbit
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: ""
resources:
requests:
storage: 1Gi
selector:
matchLabels:
type: pv-rabbitmq
# 👍 END
| Task | Imperative Command | Declarative YAML |
| Create Namespace | kubectl create namespace dev | apiVersion: v1 kind: Namespace metadata: name: dev |
| Create Pod | kubectl run nginx-pod --image=nginx:1.25 | apiVersion: v1 kind: Pod metadata: name: nginx-pod spec: containers: - name: nginx image: nginx:1.25 |
| Create Deployment | kubectl create deployment sampledeploy --image=nginx:1.25 --replicas=4 | apiVersion: apps/v1 kind: Deployment metadata: name: sampledeploy spec: replicas: 4 selector: matchLabels: app: sampledeploy template: metadata: labels: app: sampledeploy spec: containers: - name: nginx image: nginx:1.25 |
| Create ClusterIP Service | kubectl expose deployment sampledeploy --port=80 --target-port=80 | apiVersion: v1 kind: Service metadata: name: sampledeploy-service spec: selector: app: sampledeploy ports: - port: 80 targetPort: 80 |
| Create NodePort Service | kubectl expose deployment sampledeploy --type=NodePort --port=80 --target-port=80 | apiVersion: v1 kind: Service metadata: name: sampledeploy-nodeport spec: type: NodePort selector: app: sampledeploy ports: - port: 80 targetPort: 80 nodePort: 30080 |
| Create ConfigMap | kubectl create configmap app-config --from-literal=APP_MODE=production --from-literal=COLOR=blue | apiVersion: v1 kind: ConfigMap metadata: name: app-config data: APP_MODE: production COLOR: blue |
| Create Secret | kubectl create secret generic db-secret --from-literal=username=admin --from-literal=password=pass123 | apiVersion: v1 kind: Secret metadata: name: db-secret type: Opaque stringData: username: admin password: pass123 |
| Create ServiceAccount | kubectl create serviceaccount frontend-sa | apiVersion: v1 kind: ServiceAccount metadata: name: frontend-sa |
| Create Role | kubectl create role pod-reader --verb=get,list,watch --resource=pods | apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] |
| Create RoleBinding | kubectl create rolebinding pod-reader-binding --role=pod-reader --serviceaccount=default:frontend-sa | apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pod-reader-binding subjects: - kind: ServiceAccount name: frontend-sa namespace: default roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io |
| Create ClusterRole | kubectl create clusterrole node-reader --verb=get,list,watch --resource=nodes | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: node-reader rules: - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] |
| Create ClusterRoleBinding | kubectl create clusterrolebinding node-reader-binding --clusterrole=node-reader --serviceaccount=default:frontend-sa | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: node-reader-binding subjects: - kind: ServiceAccount name: frontend-sa namespace: default roleRef: kind: ClusterRole name: node-reader apiGroup: rbac.authorization.k8s.io |
| Create PVC | Usually declarative | apiVersion: v1 kind: PersistentVolumeClaim metadata: name: app-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi |
| Create StatefulSet | Usually declarative | apiVersion: apps/v1 kind: StatefulSet metadata: name: mongodb spec: serviceName: mongodb-headless replicas: 1 selector: matchLabels: app: mongodb template: metadata: labels: app: mongodb spec: containers: - name: mongodb image: mongo:6 |
| Create Job | kubectl create job test-job --image=busybox | apiVersion: batch/v1 kind: Job metadata: name: test-job spec: template: spec: containers: - name: busybox image: busybox restartPolicy: Never |
| Create CronJob | kubectl create cronjob backup-job --image=busybox --schedule="*/5 * * * *" | apiVersion: batch/v1 kind: CronJob metadata: name: backup-job spec: schedule: "*/5 * * * *" jobTemplate: spec: template: spec: containers: - name: busybox image: busybox restartPolicy: Never |
###############################
# MYSQL 8.0.x → 8.4.x UPGRADE
# CLEAN INSTALL + RESTORE METHOD
###############################
ENVIRONMENT:
- RHEL8 / Rocky Linux
- Custom datadir: /data/mysql_server
- No GTID
- Old MySQL 8.0.x
- New MySQL 8.4.x
- Using encryption/keyring
- Fresh initialize method
====================================================
STEP 1 — CHECK OLD MYSQL ENVIRONMENT
====================================================
mysql -uroot -p
SHOW PLUGINS;
SELECT TABLE_SCHEMA,TABLE_NAME,CREATE_OPTIONS
FROM information_schema.tables
WHERE CREATE_OPTIONS LIKE '%ENCRYPTION%';
grep -Ri keyring /etc/my.cnf*
====================================================
STEP 2 — TAKE FULL BACKUP
====================================================
mysqldump \
--all-databases \
--routines \
--events \
--triggers \
--single-transaction \
--hex-blob \
-u root -p > /backup/full.sql
====================================================
STEP 3 — BACKUP KEYRING (VERY IMPORTANT)
====================================================
tar -cvzf /backup/mysql-keyring.tar.gz \
/data/mysql-keyring
====================================================
STEP 4 — BACKUP CONFIG
====================================================
cp -p /etc/my.cnf /backup/
====================================================
STEP 5 — REMOVE ENCRYPTION FROM DUMP
(RECOMMENDED SAFEST METHOD)
====================================================
cp /backup/full.sql /backup/full_no_encrypt.sql
sed -i "s/ENCRYPTION='Y'/ENCRYPTION='N'/g" \
/backup/full_no_encrypt.sql
====================================================
STEP 6 — STOP MYSQL
====================================================
systemctl stop mysqld
ps -ef | grep mysqld
IF STILL RUNNING:
pkill -9 mysqld
====================================================
STEP 7 — REMOVE OLD MYSQL RPMs
====================================================
rpm -qa | grep -i mysql
dnf remove mysql*
OR
rpm -e mysql-community-server \
mysql-community-client \
mysql-community-common \
mysql-community-libs
====================================================
STEP 8 — INSTALL MYSQL 8.4 RPMs
====================================================
cd /data/pkg/mysql84/
yum install mysql-community-*.rpm
OR
rpm -ivh mysql-community-common-8.4*.rpm
rpm -ivh mysql-community-client-plugins-8.4*.rpm
rpm -ivh mysql-community-libs-8.4*.rpm
rpm -ivh mysql-community-client-8.4*.rpm
rpm -ivh mysql-community-server-8.4*.rpm
====================================================
STEP 9 — RENAME OLD DATADIR
====================================================
mv /data/mysql_server \
/data/mysql_server_80_backup
====================================================
STEP 10 — CREATE NEW DATADIR
====================================================
mkdir -p /data/mysql_server
chown -R mysql:mysql /data/mysql_server
chmod 750 /data/mysql_server
====================================================
STEP 11 — EDIT /etc/my.cnf
====================================================
REMOVE OLD KEYRING CONFIG:
#early-plugin-load=keyring_file.so
#keyring_file_data=/data/mysql-keyring/keyring
====================================================
STEP 12 — CREATE NEW KEYRING DIRECTORY
====================================================
mkdir -p /data/mysql-keyring
chown -R mysql:mysql /data/mysql-keyring
chmod 750 /data/mysql-keyring
====================================================
STEP 13 — CREATE COMPONENT CONFIG
====================================================
mkdir -p /var/lib/mysql-files
vi /var/lib/mysql-files/component_keyring_file.cnf
ADD:
{
"path": "/data/mysql-keyring/keyring",
"read_only": false
}
SAVE FILE
====================================================
STEP 14 — FIX PERMISSIONS
====================================================
chown mysql:mysql \
/var/lib/mysql-files/component_keyring_file.cnf
chmod 640 \
/var/lib/mysql-files/component_keyring_file.cnf
====================================================
STEP 15 — CREATE BOOTSTRAP FILE
(VERY IMPORTANT)
====================================================
vi /usr/sbin/mysqld.my
ADD:
{
"components": "file://component_keyring_file"
}
SAVE FILE
====================================================
STEP 16 — FIX BOOTSTRAP FILE PERMISSIONS
====================================================
chown mysql:mysql /usr/sbin/mysqld.my
chmod 640 /usr/sbin/mysqld.my
====================================================
STEP 17 — INITIALIZE MYSQL 8.4
====================================================
mysqld \
--defaults-file=/etc/my.cnf \
--initialize \
--user=mysql
====================================================
STEP 18 — START MYSQL
====================================================
systemctl start mysqld
====================================================
STEP 19 — CHECK LOGS
====================================================
journalctl -xeu mysqld
tail -f /data/mysql_server/mysqld.log
====================================================
STEP 20 — LOGIN MYSQL
====================================================
mysql -uroot -p
====================================================
STEP 21 — VERIFY KEYRING COMPONENT
====================================================
SELECT * FROM performance_schema.keyring_component_status;
IF NOT EMPTY = SUCCESS
====================================================
STEP 22 — REGISTER COMPONENT
====================================================
INSTALL COMPONENT 'file://component_keyring_file';
SELECT * FROM mysql.component;
====================================================
STEP 23 — TEST ENCRYPTION
====================================================
CREATE DATABASE testdb;
USE testdb;
CREATE TABLE t1 (
id INT
) ENCRYPTION='Y';
====================================================
STEP 24 — RESTORE DUMP
====================================================
mysql -uroot -p < /backup/full_no_encrypt.sql
====================================================
STEP 25 — VERIFY DATABASES
====================================================
SHOW DATABASES;
SELECT user,host FROM mysql.user;
====================================================
STEP 26 — OPTIONAL RE-ENABLE ENCRYPTION LATER
====================================================
ALTER TABLE table_name ENCRYPTION='Y';
====================================================
IMPORTANT NOTES
====================================================
1. NEVER DELETE:
/data/mysql-keyring
2. NEVER MIX:
old plugin + new component
3. DO NOT USE:
early-plugin-load=keyring_file.so
4. NEW MYSQL 8.4 USES:
component_keyring_file
5. MOST IMPORTANT FILE:
/usr/sbin/mysqld.my
WITHOUT IT:
- component installs
- but encryption fails
6. IF RESTORE FAILS:
- use ENCRYPTION='N'
- restore first
- re-enable later
###############################
END
###############################
Kubernetes Networking Notes
A Pod gets its own network namespace.
Inside a namespace:
are isolated from the host.
Think:
Host Network Namespace ↓ Pod Network Namespace
A veth pair is like a virtual cable.
Example:
Pod eth0 ↔ veth123
One end is inside Pod.
One end is on host.
Traffic enters/leaves Pod through veth.
A bridge works like a virtual switch.
Example:
PodA ↓ veth ↓ Bridge ↓ veth ↓ PodB
Bridge works at Layer 2.
Routing decides:
"Where should packet go next?"
Example:
DST=172.16.212.50
Kernel checks routing table.
Chooses interface.
Sends packet.
Routing happens after DNAT.
Hooks are checkpoints inside kernel.
Packet just arrived.
Used mainly for:
Packet destined for local host.
Examples:
Packet passing through host.
Examples:
Packet generated locally.
Examples:
Packet leaving system.
Used mainly for:
Change destination.
Example:
10.96.0.25 ↓ 172.16.212.50
Used by:
Change source.
Example:
172.16.212.50 ↓ 192.168.241.141
Used when returning traffic.
Dynamic SNAT.
Commonly used when Pods access outside world.
Conntrack tracks connections.
Stores:
SRC IP SRC PORT DST IP DST PORT Protocol
Example:
192.168.1.10:50000 ↓ 10.96.0.25:80
States:
NEW ESTABLISHED RELATED INVALID
Purpose:
A Service is NOT:
Service is:
A virtual IP + kube-proxy rules.
Example:
rabbitmq-service
10.96.0.25
Flow:
Pod ↓ 10.96.0.25 ↓ kube-proxy ↓ DNAT ↓ RabbitMQ Pod
Watches:
from API Server.
Creates:
Main Job:
Service IP ↓ Pod IP
Example:
10.96.0.25 ↓ 172.16.212.50
Purpose:
Name ↓ Service IP
Example:
rabbitmq ↓ 10.96.0.25
Pod resolv.conf:
nameserver 10.96.0.10
Flow:
Application ↓ DNS Query ↓ 10.96.0.10 ↓ kube-proxy ↓ CoreDNS Pod ↓ Returns Service IP
CoreDNS watches API Server.
Keeps service records in memory.
Example:
rabbitmq → 10.96.0.25
mysql → 10.96.0.50
Purpose:
Calico uses Linux networking.
Creates:
Traffic:
PodA ↓ Host Network ↓ FORWARD ↓ Calico Rules ↓ PodB
Network Policy controls:
pod communication.
Implemented by Calico.
Calico creates iptables rules.
Example:
Allow:
frontend ↓ backend
Deny:
all others
Chain = Collection of Rules.
Example:
FORWARD chain
contains many rules.
Can call another chain.
Example:
FORWARD ↓ CALICO-FORWARD ↓ Policy Rules
Think:
Main Chain ↓ Sub Chain ↓ Rules
PodA ↓ veth ↓ Host Routing ↓ Calico ↓ Worker2 ↓ PodB
PodA ↓ Service IP 10.96.0.25 ↓ kube-proxy DNAT ↓ PodB
Application
curl rabbitmq
↓
CoreDNS
↓
10.96.0.25
↓
Application connects
↓
kube-proxy
↓
RabbitMQ Pod
Ingress Object:
Routing Rules.
Example:
myrabbitmqui.com ↓ rabbitmq-service
Ingress Object does NOT receive traffic.
Actual software.
Examples:
Receives HTTP/HTTPS traffic.
Reads:
Host Header
Example:
Host: myrabbitmqui.com
↓
rabbitmq-service
Browser ↓ 443 ↓ Ingress Controller ↓ Certificate Validation ↓ Encrypted Traffic
Certificate usually stored in:
Kubernetes Secret
Browser ↓ myrabbitmqui.com ↓ DNS ↓ Worker IP / LoadBalancer ↓ Ingress Controller ↓ rabbitmq-service ↓ kube-proxy ↓ RabbitMQ Pod
Name → Service IP
Service IP → Pod IP
Can packet travel there?
How packet reaches destination
Host Name → Service
Connection Tracking
Change Destination
Change Source
Virtual IP
Actual Application
Linux To Kubernetes Networking - Complete Mental Model
Kubernetes networking is not a separate networking stack.
Kubernetes uses Linux networking.
Everything eventually becomes:
Packet → Linux Kernel → Routing → Netfilter → Socket → Application
Kubernetes components such as CoreDNS, kube-proxy, Calico, Services, and Ingress only add rules and automation on top of Linux networking.
Suppose a packet arrives from the network cable.
The Network Interface Card (NIC) receives the Ethernet frame.
The NIC places the frame into its RX Ring Buffer. This buffer is memory that allows packets to be temporarily stored until the kernel can process them.
The NIC uses DMA (Direct Memory Access) to copy the packet directly into RAM without involving the CPU. This improves performance because the CPU does not need to manually move every packet.
After copying the packet to memory, the NIC notifies the kernel.
Modern Linux uses NAPI instead of processing every packet immediately. NAPI allows packets to be processed in batches, preventing interrupt storms during high traffic.
The kernel then creates an SKB (Socket Buffer).
The SKB is the most important structure in Linux networking.
An SKB contains:
Think of an SKB as the kernel's internal representation of a packet.
The kernel first examines the Ethernet header.
The Ethernet header contains:
The EtherType tells Linux what protocol is inside the frame.
Examples:
0x0800 = IPv4 0x86DD = IPv6 0x0806 = ARP
If EtherType indicates IPv4, Linux passes the packet to the IP layer.
At the IP layer Linux examines:
Source IP Destination IP
Example:
Source IP = 192.168.1.10 Destination IP = 192.168.241.141
Before Linux decides where to send the packet, Netfilter hooks are executed.
Netfilter is a framework inside the Linux kernel.
It allows firewall, NAT and packet modification logic.
The five major hooks are:
PREROUTING INPUT FORWARD OUTPUT POSTROUTING
This hook runs before Linux makes a routing decision.
Typical uses:
Example:
Packet arrives:
Destination = 10.96.0.25
kube-proxy may change:
10.96.0.25
to
172.16.212.50
before routing happens.
INPUT is used when the packet is meant for the local machine itself.
Examples:
SSH Destination Port 22
Kubelet Destination Port 10250
The packet eventually reaches a socket owned by a process running on the host.
FORWARD is used when the packet is not for the host.
The host acts like a router.
Examples:
PodA → PodB
NodePort → Pod
Ingress Controller → Application Pod
Most Kubernetes traffic goes through FORWARD.
Used for packets created by the host itself.
Examples:
curl from the node
kubelet talking to API Server
Executed after routing decision and immediately before the packet leaves the system.
Used mainly for:
SNAT MASQUERADE
Routing determines where a packet should go.
Linux checks:
Destination IP
and consults:
Routing Table
The routing table tells Linux:
Which interface should carry the packet.
Which next hop should be used.
Routing happens after PREROUTING.
This is why DNAT must happen before routing.
If the destination IP changes, Linux needs the new destination before choosing the route.
NAT means modifying addresses.
There are three major types.
Destination NAT changes the destination address.
Example:
Destination = 10.96.0.25
becomes
Destination = 172.16.212.50
Used by:
Services NodePort Ingress
Source NAT changes the source address.
Example:
Source = 172.16.212.50
becomes
Source = 192.168.241.141
Used when traffic leaves the node.
MASQUERADE is dynamic SNAT.
Used when the external IP can change.
Commonly used when Pods access the Internet.
Conntrack is the kernel's connection tracking database.
It remembers:
Source IP Source Port Destination IP Destination Port Protocol
Suppose:
172.16.212.50:50000
connects to
10.96.0.25:80
After DNAT:
10.96.0.25
becomes
172.16.212.10
Conntrack remembers this translation.
When the response returns, conntrack ensures the reverse translation happens correctly.
Without conntrack, NAT would not work reliably.
Every Pod gets its own Network Namespace.
Inside the namespace the Pod has:
To connect the Pod to the host, Linux creates a veth pair.
Think of a veth pair as a virtual Ethernet cable.
One side exists inside the Pod.
The other side exists on the host.
Traffic leaving the Pod travels through this virtual cable.
A Service is not an application.
A Service is not a network interface.
A Service is not a process.
A Service is a virtual IP managed by kube-proxy.
Example:
rabbitmq-service
ClusterIP:
10.96.0.25
No application actually listens on 10.96.0.25.
kube-proxy intercepts packets destined for 10.96.0.25 and translates them to a real Pod IP.
kube-proxy watches the API Server.
It learns:
Services Endpoints
It then creates iptables rules.
Example:
10.96.0.25
becomes
172.16.212.50
This translation is performed using DNAT.
Therefore the primary job of kube-proxy is:
Service IP → Pod IP
Applications prefer names rather than IP addresses.
Example:
rabbitmq
instead of
10.96.0.25
CoreDNS watches the API Server and maintains DNS records in memory.
Example:
rabbitmq
→
10.96.0.25
When a Pod performs DNS lookup, CoreDNS returns the Service IP.
CoreDNS performs:
Name → Service IP
kube-proxy performs:
Service IP → Pod IP
Calico provides Pod networking and Network Policies.
Calico does not forward packets itself.
Linux performs the forwarding.
Calico programs:
Routes iptables rules Network Policies
Think:
Linux does the work.
Calico tells Linux what rules to use.
An Ingress object is only a routing rule.
Example:
myrabbitmqui.com
→
rabbitmq-service
The Ingress object itself receives no traffic.
An Ingress Controller such as NGINX reads the Ingress object and performs the routing.
NGINX receives the request, examines the Host header, and chooses the correct Service.
Browser opens:
https://myrabbitmqui.com
DNS returns:
192.168.241.141
Packet arrives at Worker Node.
NIC receives frame.
DMA copies frame into RAM.
NAPI schedules packet processing.
Kernel creates SKB.
Ethernet layer processes frame.
IP layer processes packet.
PREROUTING executes.
kube-proxy translates WorkerIP:443 to Ingress Controller Pod IP.
Routing decision occurs.
Packet traverses FORWARD chain.
Calico policy is checked.
Packet reaches NGINX Ingress Controller.
NGINX reads:
Host: myrabbitmqui.com
NGINX chooses:
rabbitmq-service
NGINX sends a new request to:
10.96.0.25
PREROUTING executes again.
kube-proxy translates:
10.96.0.25
to
172.16.212.50
Routing occurs.
Packet traverses FORWARD chain.
Packet reaches RabbitMQ Pod.
TCP delivers the packet to RabbitMQ's socket.
RabbitMQ application processes the request and sends the response back.
# Kubernetes Pod Mastery - Module 2 Notes
## Process Lifecycle, Restart Policies, Command & Args
------------------------------------------------------------
1. CONTAINER PROCESS LIFECYCLE
------------------------------------------------------------
Every container has ONE main process (PID 1).
The lifecycle of the container depends on this process.
• If the main process is running
→ Container = Running
• If the main process exits with Exit Code 0
→ Container = Completed
• If the main process exits with a non-zero Exit Code
→ Container = Error
Examples:
sleep 300
→ Running
echo "Hello Kubernetes"
→ Completed (Exit Code 0)
false
→ Error (Exit Code 1)
------------------------------------------------------------
2. RESTART POLICIES
------------------------------------------------------------
The kubelet monitors the main process of every container.
When the process exits, kubelet checks the Pod's restartPolicy before deciding whether to create a new container.
There are three restart policies.
A) restartPolicy: Never
• Never restart the container.
Exit Code 0
→ Completed
Exit Code 1
→ Error
------------------------------------------------------------
B) restartPolicy: OnFailure
• Restart only when the process exits with a non-zero Exit Code.
Exit Code 0
→ No Restart
Exit Code 1
→ Restart
------------------------------------------------------------
C) restartPolicy: Always
• Restart the container regardless of whether it exits successfully or fails.
Exit Code 0
→ Restart
Exit Code 1
→ Restart
------------------------------------------------------------
3. CrashLoopBackOff
------------------------------------------------------------
CrashLoopBackOff is NOT a restart policy.
It is a waiting state where kubelet repeatedly restarts a container and applies an exponential backoff delay before trying again.
CrashLoopBackOff can occur when:
• Application exits with Exit Code 1 repeatedly.
OR
• Application exits successfully (Exit Code 0) very quickly (for example sleep 2) while restartPolicy is Always.
The kubelet gradually increases the delay between restart attempts.
------------------------------------------------------------
4. COMMAND AND ARGS
------------------------------------------------------------
Kubernetes uses:
command
→ Executable
args
→ Arguments passed to the executable
Final execution is conceptually:
command[] + args[]
Example:
command:
- echo
- hello
args:
- kubernetes
Final execution:
echo hello kubernetes
The first element of command is the executable.
The remaining elements of command become arguments.
The args array is appended after those arguments.
------------------------------------------------------------
5. Docker ENTRYPOINT and CMD
------------------------------------------------------------
Docker images contain default startup information.
Docker ENTRYPOINT
=
Kubernetes command
Docker CMD
=
Kubernetes args
Images also store metadata like:
• ENTRYPOINT
• CMD
• ENV
• USER
• WORKDIR
• EXPOSE
• LABEL
etc.
------------------------------------------------------------
6. OVERRIDE RULES
------------------------------------------------------------
Case 1
No command
No args
Result:
Use Image ENTRYPOINT
+
Use Image CMD
------------------------------------------------------------
Case 2
Only args
Result:
Use Image ENTRYPOINT
Replace Image CMD with Pod args
------------------------------------------------------------
Case 3
Only command
Result:
Use Pod command
Ignore Image ENTRYPOINT
Ignore Image CMD
------------------------------------------------------------
Case 4
command + args
Result:
Use Pod command
Use Pod args
Ignore Image ENTRYPOINT
Ignore Image CMD
------------------------------------------------------------
7. HOW KUBERNETES STARTS A CONTAINER
------------------------------------------------------------
kubectl apply
|
V
API Server
|
V
Scheduler
|
V
Kubelet
|
V
Container Runtime (containerd / CRI-O)
|
V
OCI Runtime (runc)
|
V
Linux execve()
|
V
Container Starts
------------------------------------------------------------
8. INTERNAL FLOW
------------------------------------------------------------
The image is NEVER modified.
The image already contains metadata:
• ENTRYPOINT
• CMD
• ENV
• USER
• WORKDIR
Containerd pulls the image and reads this metadata.
Kubelet reads the Pod specification.
Kubelet sends any overrides such as:
• command
• args
• environment variables
• volume mounts
• security settings
Containerd combines:
Image defaults
+
Pod overrides
Containerd generates the OCI Runtime Specification.
Finally, runc executes the Linux process using execve().
------------------------------------------------------------
9. IMPORTANT CKA POINTS
------------------------------------------------------------
✓ Every container has one main process.
✓ Exit Code 0 = Completed.
✓ Exit Code non-zero = Error.
✓ Kubelet decides whether to restart a container.
✓ CrashLoopBackOff is a restart backoff state, not a restart policy.
✓ command specifies the executable.
✓ args specifies arguments.
✓ command overrides the image ENTRYPOINT.
✓ args overrides the image CMD.
✓ Images are read-only; Kubernetes never modifies an image.
✓ Containerd reads image metadata and combines it with kubelet's Pod configuration before starting the container.
------------------------------------------------------------
10. MY MENTAL MODEL
------------------------------------------------------------
Whenever I troubleshoot a Pod, I ask these questions:
1. What is the main process?
2. Is the process still running?
3. If it exited:
- What is the Exit Code?
4. What is the restartPolicy?
5. Is kubelet restarting the container?
6. Is the container in CrashLoopBackOff?
7. What executable (command) is Kubernetes starting?
8. What arguments (args) are being passed?
If I can answer these questions, I can usually determine why a Pod is in Running, Completed, Error, or CrashLoopBackOff.
==========================================================
MODULE 4 - HEALTH PROBES (REVISION NOTES)
==========================================================
##########################################################
1. WHY DO WE NEED PROBES?
##########################################################
A container can be:
1. Starting
2. Running but not ready for traffic
3. Running and healthy
4. Running but hung/dead internally
Kubelet cannot determine all of these states by only checking if
the process exists.
Therefore Kubernetes provides three probes:
1. Startup Probe
2. Readiness Probe
3. Liveness Probe
----------------------------------------------------------
##########################################################
2. STARTUP PROBE
##########################################################
Purpose:
--------
Checks whether an application can successfully complete startup.
Question it answers:
--------------------
"Can this application finish starting?"
Used for:
---------
- Slow starting applications
- Spring Boot
- Java applications
- Applications loading cache
- Database initialization
How it works:
-------------
Container Starts
↓
Startup Probe runs
↓
If Fail
↓
Keep checking
↓
If Success (ONLY ONCE)
↓
Startup Probe is DISABLED forever
↓
Readiness and Liveness begin
Important:
----------
• Runs only during startup.
• Runs until first success.
• After success it never runs again for that container.
• If container restarts, Startup Probe starts again because it is a NEW container instance.
Configuration:
--------------
initialDelaySeconds
periodSeconds
failureThreshold
timeoutSeconds
Maximum startup time allowed:
initialDelaySeconds +
(periodSeconds × failureThreshold)
If startup never succeeds within this time:
→ Kubelet restarts container.
----------------------------------------------------------
##########################################################
3. READINESS PROBE
##########################################################
Purpose:
--------
Determines whether a Pod should receive traffic.
Question it answers:
--------------------
"Can I serve user requests?"
If Readiness succeeds:
Ready=True
↓
EndpointSlice Controller adds Pod IP
↓
kube-proxy updates iptables/IPVS
↓
Service sends traffic
If Readiness fails:
Ready=False
↓
EndpointSlice Controller removes Pod IP
↓
kube-proxy updates iptables/IPVS
↓
Service stops sending traffic
IMPORTANT:
----------
Readiness NEVER restarts a container.
Container continues running.
Only traffic is stopped.
----------------------------------------------------------
##########################################################
4. LIVENESS PROBE
##########################################################
Purpose:
--------
Determines whether the application is still healthy.
Question it answers:
--------------------
"Am I still alive?"
If Liveness succeeds:
Nothing happens.
Application keeps running.
If Liveness fails:
FailureThreshold reached
↓
Kubelet kills container
↓
RestartPolicy checked
↓
Container restarted
Repeated failures:
Restart
↓
Restart
↓
Restart
↓
Exponential Backoff
↓
CrashLoopBackOff
IMPORTANT:
----------
Liveness DOES restart containers.
----------------------------------------------------------
##########################################################
5. EXECUTION ORDER
##########################################################
Container Starts
│
▼
Startup Probe
│
▼
Startup Success
│
├─────────────┐
▼ ▼
Readiness Liveness
│ │
▼ ▼
Traffic Restart if unhealthy
----------------------------------------------------------
##########################################################
6. KUBELET RESPONSIBILITIES
##########################################################
Kubelet:
• Runs Startup Probe
• Runs Readiness Probe
• Runs Liveness Probe
• Updates Pod Ready condition
• Restarts failed containers
• Applies Restart Policy
----------------------------------------------------------
##########################################################
7. STARTUP vs READINESS vs LIVENESS
##########################################################
Startup Probe
Purpose:
Can application finish startup?
Controls:
Startup phase only
Restarts:
YES (if startup never succeeds)
Traffic:
NO
----------------------------------------------------------
Readiness Probe
Purpose:
Can application receive traffic?
Controls:
Traffic routing
Restarts:
NO
Traffic:
YES
----------------------------------------------------------
Liveness Probe
Purpose:
Is application still healthy?
Controls:
Container restart
Restarts:
YES
Traffic:
NO
----------------------------------------------------------
##########################################################
8. STARTUP PROBE vs LIVENESS initialDelaySeconds
##########################################################
Liveness initialDelaySeconds
Meaning:
"Wait X seconds.
After that I expect application to be healthy."
Problem:
If startup takes longer than expected,
container gets killed repeatedly.
----------------------------------------------------------
Startup Probe
Meaning:
"I don't know how long startup takes.
I'll keep checking until startup succeeds
or maximum startup time expires."
This removes the need to guess startup time.
----------------------------------------------------------
##########################################################
9. CRASHLOOPBACKOFF
##########################################################
CrashLoopBackOff is NOT a Pod phase.
It is kubectl STATUS shown when:
Container repeatedly exits
AND
RestartPolicy causes restart
AND
Kubelet applies exponential restart delay.
Typical delays:
Restart immediately
↓
10 sec
↓
20 sec
↓
40 sec
↓
80 sec
↓
160 sec
↓
300 sec (maximum backoff)
After maximum delay,
Kubelet still continues retrying approximately every 300 seconds
until the Pod is deleted or the problem is fixed.
----------------------------------------------------------
##########################################################
10. PROBE TIMELINE
##########################################################
Container Starts
│
▼
Startup Probe
│
├── Fail → Keep waiting
│
└── Success
│
▼
Startup Probe Disabled
│
▼
Readiness Starts
│
▼
Pod Ready=True
│
▼
Service sends traffic
│
▼
Application hangs
│
├───────────────┐
▼ ▼
Readiness Fail Liveness Fail
│ │
▼ ▼
Stop Traffic Restart Container
----------------------------------------------------------
##########################################################
11. IMPORTANT CKA POINTS
##########################################################
✓ Startup runs only until first success.
✓ Startup runs again after every container restart.
✓ Readiness controls traffic only.
✓ Readiness NEVER restarts containers.
✓ Liveness restarts unhealthy containers.
✓ Startup disables Readiness and Liveness until startup succeeds.
✓ EndpointSlice Controller reacts to Ready=True/False.
✓ kube-proxy updates iptables/IPVS after EndpointSlice changes.
✓ Services only send traffic to Ready Pods.
✓ CrashLoopBackOff is caused by repeated restarts with exponential backoff.
==========================================================
END OF MODULE 4
==========================================================
===========================================================
CKA POD MASTERY
MODULE 4 – HEALTH PROBES (MASTER NOTES)
===========================================================
############################################################
WHY KUBERNETES NEEDS PROBES
############################################################
Kubelet can only see:
Container Process
Running?
Exit Code?
It CANNOT know:
✓ Is application still loading?
✓ Is application ready?
✓ Is application hung?
✓ Can application serve requests?
Therefore Kubernetes provides three probes.
------------------------------------------------------------
STARTUP PROBE
Question:
"Can application finish startup?"
READINESS PROBE
Question:
"Can application receive traffic?"
LIVENESS PROBE
Question:
"Is application still healthy?"
------------------------------------------------------------
Think like this:
Startup
↓
Readiness
↓
Liveness
============================================================
STARTUP PROBE
============================================================
Purpose
Protect slow-starting applications.
Examples
Spring Boot
Java Applications
Large Cache Loading
Database Migration
Machine Learning Models
Flow
Container Starts
↓
Startup Probe Runs
↓
Fail
↓
Keep Waiting
↓
Fail
↓
Keep Waiting
↓
Success
↓
Disable Startup Probe Forever
↓
Enable Readiness
↓
Enable Liveness
IMPORTANT
Startup Probe runs ONLY during startup.
Once it succeeds once,
it NEVER runs again
for THAT container.
If container restarts,
Startup Probe starts again.
WHY?
Because probes belong to
container instance.
Container #1
↓
Startup Passed
↓
Container Dies
↓
Container #2
↓
Startup runs AGAIN
============================================================
REAL EXAMPLE
============================================================
Application Startup
sleep 30
touch /tmp/startup-ok
touch /tmp/ready
sleep 300
Startup checks
cat /tmp/startup-ok
Timeline
0 sec
Container Starts
↓
Startup Probe
↓
Fail
↓
Fail
↓
touch /tmp/startup-ok
↓
Startup Success
↓
Startup Disabled
↓
Readiness Starts
↓
Liveness Starts
============================================================
READINESS PROBE
============================================================
Purpose
Should this Pod receive traffic?
If Ready=True
↓
API Server updated
↓
EndpointSlice Controller
↓
Pod IP Added
↓
kube-proxy
↓
iptables/IPVS updated
↓
Traffic Starts
-------------------------------------
If Ready=False
↓
API Server updated
↓
EndpointSlice Controller
↓
Remove Pod IP
↓
kube-proxy updates
↓
NO TRAFFIC
IMPORTANT
Readiness NEVER
kills
restarts
or recreates container.
It ONLY controls traffic.
============================================================
REAL EXAMPLE
============================================================
Application Running
↓
rm /tmp/ready
↓
Readiness Probe
↓
Fail
↓
Ready=False
↓
EndpointSlice removes Pod
↓
Service Stops Sending Traffic
Container STILL RUNNING
============================================================
LIVENESS PROBE
============================================================
Purpose
Is application still alive?
If Healthy
↓
Nothing
If Unhealthy
↓
Failure Threshold Reached
↓
Kubelet
↓
Kill Container
↓
Restart Policy
↓
Restart Container
============================================================
REAL EXAMPLE
============================================================
Application
touch /tmp/ready
↓
Later
rm /tmp/ready
↓
Liveness
Fail
↓
Fail
↓
Fail
↓
Kubelet Restarts Container
============================================================
STARTUP vs READINESS vs LIVENESS
============================================================
Startup
Protect Startup
Runs once
Can Restart
No Traffic Control
-------------------------------------
Readiness
Traffic Control
Runs continuously
Never Restarts
Controls EndpointSlice
-------------------------------------
Liveness
Health Check
Runs continuously
Restarts Container
No Traffic Control
============================================================
INITIAL DELAY
============================================================
Startup initialDelay
Wait before checking startup.
Useful when app
ALWAYS needs
some minimum time.
------------------------------------------------
Liveness initialDelay
Wait before checking health.
Problem
You must GUESS startup time.
If app starts slower
↓
Liveness kills container.
This is WHY Startup Probe exists.
============================================================
FAILURE THRESHOLD
============================================================
Example
periodSeconds = 10
failureThreshold = 3
Three consecutive failures
↓
Action
Readiness
↓
NotReady
Liveness
↓
Restart
Startup
↓
Startup Failed
Restart Container
============================================================
CRASHLOOPBACKOFF
============================================================
CrashLoopBackOff
means
Container keeps restarting.
Kubelet applies
Exponential Delay
Typical
Restart
↓
10 sec
↓
20 sec
↓
40 sec
↓
80 sec
↓
160 sec
↓
300 sec
↓
300 sec
↓
300 sec
continues forever
until fixed.
============================================================
COMPLETE TIMELINE
============================================================
Container Starts
↓
Startup Probe
↓
Success
↓
Readiness
↓
Ready=True
↓
Service sends traffic
↓
Application hangs
↓
Readiness fails
↓
Traffic Stops
↓
Liveness fails
↓
Restart
↓
NEW CONTAINER
↓
Startup Probe AGAIN
============================================================
MY MISTAKES (VERY IMPORTANT)
============================================================
Mistake 1
I wrote
command:
- sleep
- 30
- touch /tmp/file
Wrong
Reason
command is NOT shell script.
First value
Executable
Remaining values
Arguments
Correct
command:
- sh
- -c
- |
sleep 30
touch /tmp/file
------------------------------------------------
Mistake 2
I thought
CrashLoopBackOff
means Pod stopped.
Reality
Pod can show
Running
while kubelet
is retrying container.
CrashLoopBackOff
is restart state,
NOT pod phase.
------------------------------------------------
Mistake 3
I thought
Readiness restart container.
Reality
Readiness NEVER restarts.
Only removes traffic.
------------------------------------------------
Mistake 4
I thought
Startup never runs again.
Reality
Startup runs again
AFTER EVERY CONTAINER RESTART.
------------------------------------------------
Mistake 5
Confused with seconds.
Solution
STOP calculating time.
Instead
Draw Timeline.
Example
Container
↓
Startup
↓
Ready
↓
Traffic
↓
Readiness Fail
↓
Traffic Stops
↓
Liveness Fail
↓
Restart
Much easier than counting seconds.
============================================================
CKA EXAM POINTS
============================================================
✓ Startup protects startup.
✓ Readiness controls traffic.
✓ Liveness restarts containers.
✓ Startup disables
Readiness & Liveness
until startup succeeds.
✓ Readiness modifies
EndpointSlice.
✓ kube-proxy updates
iptables/IPVS.
✓ Services send traffic
ONLY to Ready Pods.
✓ Startup belongs to
container instance.
✓ CrashLoopBackOff
uses exponential backoff.
============================================================
END OF MODULE 4
============================================================
POD MASTERY
│
├── 1. Pod Fundamentals
│ ├── Pod Concept
│ ├── Smallest Deployable Unit
│ ├── Pod vs Container
│ ├── Multi-Container Pod
│ ├── Shared Network Namespace
│ ├── Shared Storage
│ └── Shared Pod Resources
│
├── 2. Pod Lifecycle
│ ├── Pod Phases
│ │ ├── Pending
│ │ ├── Running
│ │ ├── Succeeded
│ │ ├── Failed
│ │ └── Unknown
│ │
│ ├── Container States
│ │ ├── Waiting
│ │ ├── Running
│ │ └── Terminated
│ │
│ ├── Restart Policy
│ │ ├── Always
│ │ ├── OnFailure
│ │ └── Never
│ │
│ ├── Kubelet Container Monitoring
│ └── Container Exit Status
│
├── 3. Pod Initialization & Containers
│ ├── Init Containers
│ ├── Init Container Ordering
│ ├── Init Container Success/Failure
│ ├── Multiple Init Containers
│ ├── Multi-Container Pods
│ ├── Container Communication
│ └── Sidecar Concept
│
├── 4. Pod Networking
│ ├── Pod Network Namespace
│ ├── Pod IP
│ ├── Container-to-Container Communication
│ ├── localhost Inside Pod
│ ├── Network Namespace Sharing
│ ├── veth Pair
│ ├── CNI
│ └── Pod Network Flow
│
├── 5. Pod Storage
│ ├── Volume Concept
│ ├── Volume Lifecycle
│ │
│ ├── emptyDir
│ │ ├── Pod Lifetime
│ │ ├── Container Restart
│ │ └── Shared Between Containers
│ │
│ ├── hostPath
│ │ ├── Node Filesystem
│ │ ├── Node Dependency
│ │ └── DirectoryOrCreate
│ │
│ ├── ConfigMap Volume
│ ├── Secret Volume
│ │ ├── data
│ │ ├── stringData
│ │ └── Base64 Encoding
│ │
│ ├── Projected Volume
│ │ ├── ConfigMap
│ │ ├── Secret
│ │ ├── Downward API
│ │ └── ServiceAccount Token
│ │
│ └── Downward API
│ ├── Pod Metadata
│ ├── Pod Name
│ ├── Namespace
│ └── Pod Fields
│
├── 6. Pod Resources
│ ├── Resource Requests
│ ├── Resource Limits
│ ├── CPU
│ ├── Memory
│ ├── Ephemeral Storage
│ ├── Scheduler Uses Requests
│ ├── Limits Enforcement
│ ├── QoS Classes
│ │ ├── Guaranteed
│ │ ├── Burstable
│ │ └── BestEffort
│ ├── OOMKilled
│ └── Resource Pressure
│
├── 7. Pod Health & Probes
│ ├── Why Probes Are Needed
│ ├── Liveness Probe
│ ├── Readiness Probe
│ ├── Startup Probe
│ ├── Probe Types
│ │ ├── HTTP
│ │ ├── TCP
│ │ ├── Exec
│ │ └── GRPC
│ ├── Probe Timing
│ │ ├── initialDelaySeconds
│ │ ├── periodSeconds
│ │ ├── timeoutSeconds
│ │ ├── failureThreshold
│ │ └── successThreshold
│ └── Probe Failure Behavior
│
├── 8. Pod Identity & RBAC
│ ├── ServiceAccount
│ ├── Default ServiceAccount
│ ├── serviceAccountName
│ ├── ServiceAccount Token
│ ├── automountServiceAccountToken
│ ├── Authentication
│ ├── Authorization
│ ├── Role
│ ├── ClusterRole
│ ├── RoleBinding
│ ├── ClusterRoleBinding
│ ├── Namespace Scope
│ └── Cluster-Wide Scope
│
├── 9. Pod SecurityContext
│ ├── Why SecurityContext
│ ├── Pod-Level vs Container-Level
│ ├── runAsUser
│ ├── runAsGroup
│ ├── supplementalGroups
│ ├── fsGroup
│ ├── runAsNonRoot
│ ├── readOnlyRootFilesystem
│ ├── Linux Capabilities
│ │ ├── add
│ │ └── drop
│ ├── privileged
│ ├── allowPrivilegeEscalation
│ └── seccompProfile
│ ├── RuntimeDefault
│ ├── Unconfined
│ └── Localhost
│
├── 10. Pod Security Standards
│ ├── PSS
│ │ ├── Privileged
│ │ ├── Baseline
│ │ └── Restricted
│ │
│ ├── Pod Security Admission
│ ├── Admission Phase
│ ├── Namespace Labels
│ ├── enforce
│ ├── warn
│ └── audit
│
├── 11. Pod Scheduling
│ ├── Scheduler Flow
│ ├── nodeName
│ ├── nodeSelector
│ ├── Node Affinity
│ ├── Pod Affinity
│ ├── Pod Anti-Affinity
│ ├── topologyKey
│ ├── Taints
│ │ ├── NoSchedule
│ │ ├── PreferNoSchedule
│ │ └── NoExecute
│ └── Tolerations
│ ├── Equal
│ └── Exists
│
├── 12. Pod Termination
│ ├── Pod Deletion
│ ├── Termination Grace Period
│ ├── preStop Hook
│ ├── SIGTERM
│ ├── SIGKILL
│ └── Graceful Shutdown
│
├── 13. Pod State & Conditions
│ ├── Pod Phase
│ ├── Pod Conditions
│ │ ├── PodScheduled
│ │ ├── Initialized
│ │ ├── ContainersReady
│ │ └── Ready
│ ├── Container State
│ └── Phase vs Condition vs Container State
│
├── 14. Pod DNS
│ ├── Pod Hostname
│ ├── /etc/hosts
│ ├── /etc/resolv.conf
│ ├── CoreDNS
│ ├── DNS Search Domains
│ ├── ndots
│ ├── dnsPolicy
│ └── hostNetwork Interaction
│
├── 15. Pod Eviction & Node Pressure
│ ├── MemoryPressure
│ ├── DiskPressure
│ ├── PIDPressure
│ ├── Ephemeral Storage Pressure
│ ├── Kubelet Eviction
│ ├── QoS and Eviction
│ └── OOMKilled vs Eviction
│
├── 16. Static Pods
│ ├── Static Pod Concept
│ ├── Kubelet Managed
│ ├── Static Pod Manifest
│ ├── /etc/kubernetes/manifests/
│ ├── Static Pod vs Normal Pod
│ └── Control Plane Static Pods
│
└── 17. Pod Troubleshooting & Final Mastery
├── Pending
├── CrashLoopBackOff
├── ImagePullBackOff
├── CreateContainerConfigError
├── ContainerCreating
├── OOMKilled
├── Permission Denied
├── Volume Mount Failure
├── ConfigMap/Secret Failure
├── Probe Failure
├── SecurityContext Failure
├── DNS Failure
├── Scheduling Failure
├── Node Pressure / Eviction
└── Final Pod Mastery Lab & Test
Calico Pod Communication
## 🧱 1. The Core Infrastructure Components
Before any traffic moves, Calico and Kubernetes set up the landscape:
* The veth Pairs: For every Pod created, Calico creates a virtual ethernet cable. One end sits in the Pod's network namespace; the other end sits in the host namespace (named caliXXXXX).
* The Map (Routing Table): Calico’s agent (Felix) pre-populates the Linux kernel’s native routing table (ip route). It writes rules mapping every Pod IP to its specific caliXXXXX interface.
* The Security Guard (Netfilter/iptables): Calico and kube-proxy inject custom sub-chains (starting with cali- and KUBE-) directly into the kernel's native Netfilter hooks.
------------------------------
## 🕒 2. The Strict Order of Operations
When a packet travels from Pod 1 to a Service VIP, it triggers Netfilter hooks and routing decisions in a strict, unchangeable timeline:
## Hook 1: PREROUTING (The Entry Checkpoint)
* kube-proxy hits first: It intercepts the virtual Service VIP and executes DNAT, rewriting the destination address to a Real Backend Pod IP.
* Calico hits immediately after: It runs its cali-PREROUTING chain. It inspects the packet after DNAT and applies Ingress Network Policies.
* Action: It either permits the packet or drops it instantly to save CPU cycles.
## Step 2: The Routing Decision (The Intersection)
* The Linux Kernel takes over: The kernel looks at the packet's destination (the Real Pod IP).
* The Map Lookup: It references the routing table that Calico pre-configured.
* Action: The kernel matches the Pod IP and determines the exact exit interface (caliXXXXX for same-host, or a physical NIC like eth0 for cross-host).
## Hook 3: FORWARD (The Transit Lane)
* The Context: Because the packet is crossing from one network interface namespace to another, the kernel routes it through the FORWARD hook.
* Calico hits here: It runs its cali-FORWARD chain to validate transit safety.
* Action: It double-checks that Pod 1 is allowed to talk to Pod 2. If a policy blocks it, the packet is killed here.
## Hook 4: POSTROUTING (The Exit Gate)
* The Context: The packet has been routed and cleared by security. It is sitting at the exit interface, ready to leave.
* Calico/kube-proxy hit here: They run the cali-POSTROUTING chain to enforce Egress Network Policies.
* Action: If the packet is exiting the cluster to the external internet, SNAT (Masquerading) is applied at the very last second, rewriting the source IP to the Node's IP so replies can find their way back.
------------------------------
## 🎯 3. Visual Execution Flow
[ Packet Leaves Pod 1 via veth ] ──► Enters Host Namespace
│
▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│ KERNEL HOOKS & CALICO INTERACTION │
│ │
│ 1. PREROUTING HOOK │
│ ├──► kube-proxy: DNAT (VIP ──► Real Pod 2 IP) │
│ └──► Calico: Checks Ingress Policy ──► [ALLOW / DROP] │
│ │
│ 2. ROUTING DECISION │
│ └──► Linux Kernel: Reads Calico's pre-written routing table │
│ └──► Destination matched ──► Target Interface locked in │
│ │
│ 3. FORWARD HOOK │
│ └──► Calico: Checks Transit/Inter-pod Policy ──► [ALLOW / DROP] │
│ │
│ 4. POSTROUTING HOOK │
│ ├──► Calico: Checks Egress Policy ──► [ALLOW / DROP] │
│ └──► kube-proxy/Calico: Applies SNAT (only if traffic leaves node/cluster) │
└─────────────────────────────────────────────────────────────────────────────────1
│
▼
[ Packet Enters Target veth ] ──► Arrives safely at Pod 2!
------------------------------
## 💡 4. The Golden Rules to Remember
1. Linux Kernel does the driving: Calico never routes a packet mid-flight. The kernel does the routing based on a map Calico drew ahead of time.
2. Calico is the firewall: Calico's primary job inside the Netfilter hooks is executing immediate Allow/Deny decisions based on your Network Policies.
3. DNAT changes the target, Routing finds the path, SNAT changes the source: DNAT happens first so the kernel knows where to route the packet. SNAT happens last so it doesn't mess up the routing decision or Calico's policy evaluation. [1]
Now that you have the complete summary, where would you like to go next?
* We can look at how this changes if you switch Calico to VXLAN/IPIP Overlay mode instead of native routing.
* We can look at how Calico's eBPF mode deletes all of these iptables hooks entirely for faster networking.