Rabbitmq Cluster On Kubernetes (Complete Setup + Troubleshooting Guide)

by abhilashthale - April 26, 2026 coding

# 🚀 RabbitMQ Cluster on Kubernetes (Complete Setup + Troubleshooting Guide)

---

# 📌 Objective

Deploy a **3-node RabbitMQ Cluster** on Kubernetes with:

* High Availability
* Persistent Storage (NFS)
* Auto Clustering
* Management UI
* Application connectivity (Tomcat)

---

# 🏗️ Components Created

## 1. Persistent Volumes (NFS)

We created 3 PVs:

* pv-rabbitmq1
* pv-rabbitmq2
* pv-rabbitmq3

Each mapped to:

```text
/data/nfsshared/rabbitmq-pv1
/data/nfsshared/rabbitmq-pv2
/data/nfsshared/rabbitmq-pv3
```

Used:

```yaml
accessModes: ReadWriteOnce
```

👉 Ensures **1 pod = 1 storage**

---

## 2. ConfigMap

Contains:

### enabled_plugins

```erlang
[rabbitmq_management,rabbitmq_peer_discovery_k8s].
```

### rabbitmq.conf

```ini
cluster_formation.peer_discovery_backend = k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = service-rabbitmq-headless
cluster_formation.k8s.hostname_suffix = .service-rabbitmq-headless.default.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
queue_master_locator=min-masters
```

👉 Enables **auto clustering using Kubernetes**

---

## 3. RBAC (CRITICAL)

```yaml
ServiceAccount → rabbitmq
Role → access pods, endpoints
RoleBinding → bind both
```

👉 Required because:

```text
RabbitMQ calls Kubernetes API → needs permission
```

---

## 4. Headless Service

```yaml
name: service-rabbitmq-headless
clusterIP: None
publishNotReadyAddresses: true
```

👉 Enables DNS like:

```text
rabbitmq-0.service-rabbitmq-headless
```

---

## 5. NodePort Service (UI)

```yaml
port: 15672
nodePort: 30072
```

👉 Access UI:

```text
http://<NodeIP>:30072
```

---

## 6. ClusterIP Service (App)

```yaml
name: rabbitmq-svc
port: 5672
```

👉 Used by:

```text
Tomcat → rabbitmq-svc:5672
```

---

## 7. StatefulSet

Key points:

```yaml
serviceName: service-rabbitmq-headless
replicas: 3
```

### ENV:

```yaml
RABBITMQ_DEFAULT_USER=admin
RABBITMQ_DEFAULT_PASS=admin
RABBITMQ_ERLANG_COOKIE=mysecretcookie
RABBITMQ_USE_LONGNAME=true
```

### Volumes:

* PVC → /var/lib/rabbitmq
* ConfigMap → rabbitmq.conf + plugins

👉 Ensures:

* Stable identity
* Persistent data
* Config-driven clustering

---

# ⚙️ FINAL EXECUTION ORDER (VERY IMPORTANT)

👉 Always follow this order:

```bash
kubectl apply -f pv-rabbit-01.yaml
kubectl apply -f pv-rabbit-02.yaml
kubectl apply -f pv-rabbit-03.yaml

kubectl apply -f rbac-rabbitmq.yaml

kubectl apply -f configmap-rabbit.yaml

kubectl apply -f service-rabbitmq-headless.yaml
kubectl apply -f service-rabbitmq-svc.yaml
kubectl apply -f service-rabbitmq-nodeport.yaml

kubectl apply -f StatefulSet-rabbitmq.yaml
```

---

# 🔥 TROUBLESHOOTING JOURNEY

---

## ❌ Issue 1: DNS Not Working

Problem:

```text
rabbitmq-1 not resolving
```

Fix:

```yaml
publishNotReadyAddresses: true
```

---

## ❌ Issue 2: Service Name Mismatch

Problem:

```text
rabbitmq-headless vs service-rabbitmq-headless
```

Fix:

```text
Must match EXACTLY
```

---

## ❌ Issue 3: No rabbitmq.conf

Fix:
Added clustering config

---

## ❌ Issue 4: 403 Error (CRITICAL)

Log:

```text
Failed to fetch nodes from Kubernetes API: 403
```

Fix:
Added RBAC

---

## ❌ Issue 5: Short vs Long Names

Error:

```text
epmd nxdomain
```

Fix:

```yaml
RABBITMQ_USE_LONGNAME=true
```

---

## ❌ Issue 6: Cluster Join Failure

Error:

```text
tables_not_present
mnesia_not_running
```

👉 Root cause:

```text
Pods not ready at same time (timing issue)
```

---

## ❌ Issue 7: Cluster Not Forming

Final log:

```text
Starting as a blank standalone node
```

👉 Reason:

```text
Retry failed → node becomes standalone
```

---

# 🧠 WHY THIS HAPPENS

RabbitMQ:

```text
Cluster formation happens ONLY at startup
```

If peers not ready → join fails

---

# 🔧 FINAL FIXES APPLIED

* Enabled RBAC ✅
* Enabled longnames ✅
* Fixed serviceName ✅
* Fixed DNS ✅
* Added retry logic ✅
* Restarted pods cleanly ✅

---

# 📊 FINAL RESULT

```bash
rabbitmqctl cluster_status
```

Output:

```text
rabbit@rabbitmq-0
rabbit@rabbitmq-1
rabbit@rabbitmq-2
```

---

# 🎯 WHAT WE ACHIEVED

✅ 3-node RabbitMQ cluster
✅ Auto discovery via Kubernetes
✅ Persistent storage
✅ UI access
✅ App connectivity
✅ HA-ready setup

---

# ⚠️ ALTERNATIVES

| Approach | Result |
| -------------- | ------------------------ |
| No RBAC | No clustering ❌ |
| Manual join | Works but not stable ⚠️ |
| Classic config | Static, not scalable ❌ |
| Helm chart | Best production option ✅ |

---

# 🧠 FINAL LEARNING

* Kubernetes = dynamic → needs API
* RabbitMQ = startup-based clustering
* RBAC = mandatory
* Headless service = must
* Longnames = required
* Timing = critical

---

# 🚀 NEXT STEPS

* Create quorum queues
* Test failover (kill pod)
* Connect Tomcat producer/consumer
* Monitor cluster

---

# 📌 FINAL CONCLUSION

You successfully built a **production-grade RabbitMQ cluster on Kubernetes**
and solved real-world issues like:

* DNS
* RBAC
* Clustering
* Node naming
* Startup timing

---

==================================configmap-rabbit.yaml========================================
apiVersion: v1
kind: ConfigMap
metadata:
name: configmap-rabbit
labels:
type: configmap-rabbit
data:

enabled_plugins: |
[rabbitmq_management,rabbitmq_peer_discovery_k8s].

rabbitmq.conf: |
cluster_formation.peer_discovery_backend = k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = service-rabbitmq-headless
cluster_formation.k8s.hostname_suffix = .service-rabbitmq-headless.default.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
queue_master_locator=min-masters
==================================pv-rabbit-01.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq1
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain

mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv1
readOnly: false

==================================pv-rabbit-02.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq2
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain

mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv2
readOnly: false

==================================pv-rabbit-03.yaml========================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rabbitmq3
labels:
type: pv-rabbitmq
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain

mountOptions:
- sec=sys
- nfsvers=4.1
- hard
nfs:
server: controlnode
path: /data/nfsshared/rabbitmq-pv3
readOnly: false

==================================rbac-rabbitmq.yaml========================================
apiVersion: v1
kind: ServiceAccount
metadata:
name: rabbitmq
namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rabbitmq
namespace: default
rules:
- apiGroups: [""]
resources:
- endpoints
- pods
verbs:
- get
- list
- watch

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rabbitmq
namespace: default
subjects:
- kind: ServiceAccount
name: rabbitmq
namespace: default
roleRef:
kind: Role
name: rabbitmq
apiGroup: rbac.authorization.k8s.io
==================================service-rabbitmq-headless.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: service-rabbitmq-headless
labels:
type: service-rabbitmq-headless
spec:
clusterIP: None
publishNotReadyAddresses: true
selector:
app: rabbitmq
ports:
- name: amqp
port: 5672
- name: management
port: 15672
- name: epmd
port: 4369
- name: cluster-rpc
port: 25672
==================================service-rabbitmq-nodeport.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-nodeport
labels:
type: rabbitmq-nodeport
spec:
type: NodePort
selector:
app: rabbitmq
ports:
- name: management
port: 15672
targetPort: 15672
nodePort: 30072

==================================service-rabbitmq-svc.yaml========================================
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-svc
labels:
type: rabbitmq-svc
spec:
type: ClusterIP
selector:
app: rabbitmq
ports:
- name: amqp
port: 5672
targetPort: 5672

==================================StatefulSet-rabbitmq.yaml========================================
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rabbitmq
labels:
type: rabbitmq
spec:
serviceName: service-rabbitmq-headless
replicas: 3
selector:
matchLabels:
app: rabbitmq
template:
metadata:
labels:
app: rabbitmq
spec:
serviceAccountName: rabbitmq
containers:
- name: rabbitmq
image: rabbitmq:3.12-management
ports:
- containerPort: 5672
- containerPort: 15672
env:
- name: RABBITMQ_DEFAULT_USER
value: "admin"
- name: RABBITMQ_DEFAULT_PASS
value: "admin"
- name: RABBITMQ_ERLANG_COOKIE
value: "mysecretcookie"
- name: RABBITMQ_USE_LONGNAME
value: "true"
volumeMounts:
- name: data
mountPath: /var/lib/rabbitmq
- name: config
mountPath: /etc/rabbitmq/enabled_plugins
subPath: enabled_plugins
- name: rabbitconf
mountPath: /etc/rabbitmq/rabbitmq.conf
subPath: rabbitmq.conf

volumes:
- name: config
configMap:
name: configmap-rabbit
- name: rabbitconf
configMap:
name: configmap-rabbit

volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: ""
resources:
requests:
storage: 1Gi
selector:
matchLabels:
type: pv-rabbitmq

# 👍 END

Kubernetes Imperative Vs Declarative

by abhilashthale - May 10, 2026 coding

Task	Imperative Command	Declarative YAML
Create Namespace	kubectl create namespace dev	apiVersion: v1 kind: Namespace metadata: name: dev
Create Pod	kubectl run nginx-pod --image=nginx:1.25	apiVersion: v1 kind: Pod metadata: name: nginx-pod spec: containers: - name: nginx image: nginx:1.25
Create Deployment	kubectl create deployment sampledeploy --image=nginx:1.25 --replicas=4	apiVersion: apps/v1 kind: Deployment metadata: name: sampledeploy spec: replicas: 4 selector: matchLabels: app: sampledeploy template: metadata: labels: app: sampledeploy spec: containers: - name: nginx image: nginx:1.25
Create ClusterIP Service	kubectl expose deployment sampledeploy --port=80 --target-port=80	apiVersion: v1 kind: Service metadata: name: sampledeploy-service spec: selector: app: sampledeploy ports: - port: 80 targetPort: 80
Create NodePort Service	kubectl expose deployment sampledeploy --type=NodePort --port=80 --target-port=80	apiVersion: v1 kind: Service metadata: name: sampledeploy-nodeport spec: type: NodePort selector: app: sampledeploy ports: - port: 80 targetPort: 80 nodePort: 30080
Create ConfigMap	kubectl create configmap app-config --from-literal=APP_MODE=production --from-literal=COLOR=blue	apiVersion: v1 kind: ConfigMap metadata: name: app-config data: APP_MODE: production COLOR: blue
Create Secret	kubectl create secret generic db-secret --from-literal=username=admin --from-literal=password=pass123	apiVersion: v1 kind: Secret metadata: name: db-secret type: Opaque stringData: username: admin password: pass123
Create ServiceAccount	kubectl create serviceaccount frontend-sa	apiVersion: v1 kind: ServiceAccount metadata: name: frontend-sa
Create Role	kubectl create role pod-reader --verb=get,list,watch --resource=pods	apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"]
Create RoleBinding	kubectl create rolebinding pod-reader-binding --role=pod-reader --serviceaccount=default:frontend-sa	apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pod-reader-binding subjects: - kind: ServiceAccount name: frontend-sa namespace: default roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io
Create ClusterRole	kubectl create clusterrole node-reader --verb=get,list,watch --resource=nodes	apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: node-reader rules: - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"]
Create ClusterRoleBinding	kubectl create clusterrolebinding node-reader-binding --clusterrole=node-reader --serviceaccount=default:frontend-sa	apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: node-reader-binding subjects: - kind: ServiceAccount name: frontend-sa namespace: default roleRef: kind: ClusterRole name: node-reader apiGroup: rbac.authorization.k8s.io
Create PVC	Usually declarative	apiVersion: v1 kind: PersistentVolumeClaim metadata: name: app-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi
Create StatefulSet	Usually declarative	apiVersion: apps/v1 kind: StatefulSet metadata: name: mongodb spec: serviceName: mongodb-headless replicas: 1 selector: matchLabels: app: mongodb template: metadata: labels: app: mongodb spec: containers: - name: mongodb image: mongo:6
Create Job	kubectl create job test-job --image=busybox	apiVersion: batch/v1 kind: Job metadata: name: test-job spec: template: spec: containers: - name: busybox image: busybox restartPolicy: Never
Create CronJob	kubectl create cronjob backup-job --image=busybox --schedule="/5 * * *"	apiVersion: batch/v1 kind: CronJob metadata: name: backup-job spec: schedule: "/5 * * *" jobTemplate: spec: template: spec: containers: - name: busybox image: busybox restartPolicy: Never

Mysql 8.0.X To 8.4.X Upgrade Runbook (Clean Install + Keyring Migration + Encryption Testing)

by abhilashthale - May 19, 2026 coding

###############################
# MYSQL 8.0.x → 8.4.x UPGRADE
# CLEAN INSTALL + RESTORE METHOD
###############################

ENVIRONMENT:
- RHEL8 / Rocky Linux
- Custom datadir: /data/mysql_server
- No GTID
- Old MySQL 8.0.x
- New MySQL 8.4.x
- Using encryption/keyring
- Fresh initialize method

====================================================
STEP 1 — CHECK OLD MYSQL ENVIRONMENT
====================================================

mysql -uroot -p

SHOW PLUGINS;

SELECT TABLE_SCHEMA,TABLE_NAME,CREATE_OPTIONS
FROM information_schema.tables
WHERE CREATE_OPTIONS LIKE '%ENCRYPTION%';

grep -Ri keyring /etc/my.cnf*

====================================================
STEP 2 — TAKE FULL BACKUP
====================================================

mysqldump \
--all-databases \
--routines \
--events \
--triggers \
--single-transaction \
--hex-blob \
-u root -p > /backup/full.sql

====================================================
STEP 3 — BACKUP KEYRING (VERY IMPORTANT)
====================================================

tar -cvzf /backup/mysql-keyring.tar.gz \
/data/mysql-keyring

====================================================
STEP 4 — BACKUP CONFIG
====================================================

cp -p /etc/my.cnf /backup/

====================================================
STEP 5 — REMOVE ENCRYPTION FROM DUMP
(RECOMMENDED SAFEST METHOD)
====================================================

cp /backup/full.sql /backup/full_no_encrypt.sql

sed -i "s/ENCRYPTION='Y'/ENCRYPTION='N'/g" \
/backup/full_no_encrypt.sql

====================================================
STEP 6 — STOP MYSQL
====================================================

systemctl stop mysqld

ps -ef | grep mysqld

IF STILL RUNNING:

pkill -9 mysqld

====================================================
STEP 7 — REMOVE OLD MYSQL RPMs
====================================================

rpm -qa | grep -i mysql

dnf remove mysql*

OR

rpm -e mysql-community-server \
mysql-community-client \
mysql-community-common \
mysql-community-libs

====================================================
STEP 8 — INSTALL MYSQL 8.4 RPMs
====================================================

cd /data/pkg/mysql84/

yum install mysql-community-*.rpm

OR

rpm -ivh mysql-community-common-8.4*.rpm
rpm -ivh mysql-community-client-plugins-8.4*.rpm
rpm -ivh mysql-community-libs-8.4*.rpm
rpm -ivh mysql-community-client-8.4*.rpm
rpm -ivh mysql-community-server-8.4*.rpm

====================================================
STEP 9 — RENAME OLD DATADIR
====================================================

mv /data/mysql_server \
/data/mysql_server_80_backup

====================================================
STEP 10 — CREATE NEW DATADIR
====================================================

mkdir -p /data/mysql_server

chown -R mysql:mysql /data/mysql_server

chmod 750 /data/mysql_server

====================================================
STEP 11 — EDIT /etc/my.cnf
====================================================

REMOVE OLD KEYRING CONFIG:

#early-plugin-load=keyring_file.so
#keyring_file_data=/data/mysql-keyring/keyring

====================================================
STEP 12 — CREATE NEW KEYRING DIRECTORY
====================================================

mkdir -p /data/mysql-keyring

chown -R mysql:mysql /data/mysql-keyring

chmod 750 /data/mysql-keyring

====================================================
STEP 13 — CREATE COMPONENT CONFIG
====================================================

mkdir -p /var/lib/mysql-files

vi /var/lib/mysql-files/component_keyring_file.cnf

ADD:

{
"path": "/data/mysql-keyring/keyring",
"read_only": false
}

SAVE FILE

====================================================
STEP 14 — FIX PERMISSIONS
====================================================

chown mysql:mysql \
/var/lib/mysql-files/component_keyring_file.cnf

chmod 640 \
/var/lib/mysql-files/component_keyring_file.cnf

====================================================
STEP 15 — CREATE BOOTSTRAP FILE
(VERY IMPORTANT)
====================================================

vi /usr/sbin/mysqld.my

ADD:

{
"components": "file://component_keyring_file"
}

SAVE FILE

====================================================
STEP 16 — FIX BOOTSTRAP FILE PERMISSIONS
====================================================

chown mysql:mysql /usr/sbin/mysqld.my

chmod 640 /usr/sbin/mysqld.my

====================================================
STEP 17 — INITIALIZE MYSQL 8.4
====================================================

mysqld \
--defaults-file=/etc/my.cnf \
--initialize \
--user=mysql

====================================================
STEP 18 — START MYSQL
====================================================

systemctl start mysqld

====================================================
STEP 19 — CHECK LOGS
====================================================

journalctl -xeu mysqld

tail -f /data/mysql_server/mysqld.log

====================================================
STEP 20 — LOGIN MYSQL
====================================================

mysql -uroot -p

====================================================
STEP 21 — VERIFY KEYRING COMPONENT
====================================================

SELECT * FROM performance_schema.keyring_component_status;

IF NOT EMPTY = SUCCESS

====================================================
STEP 22 — REGISTER COMPONENT
====================================================

INSTALL COMPONENT 'file://component_keyring_file';

SELECT * FROM mysql.component;

====================================================
STEP 23 — TEST ENCRYPTION
====================================================

CREATE DATABASE testdb;

USE testdb;

CREATE TABLE t1 (
id INT
) ENCRYPTION='Y';

====================================================
STEP 24 — RESTORE DUMP
====================================================

mysql -uroot -p < /backup/full_no_encrypt.sql

====================================================
STEP 25 — VERIFY DATABASES
====================================================

SHOW DATABASES;

SELECT user,host FROM mysql.user;

====================================================
STEP 26 — OPTIONAL RE-ENABLE ENCRYPTION LATER
====================================================

ALTER TABLE table_name ENCRYPTION='Y';

====================================================
IMPORTANT NOTES
====================================================

1. NEVER DELETE:
/data/mysql-keyring

2. NEVER MIX:
old plugin + new component

3. DO NOT USE:
early-plugin-load=keyring_file.so

4. NEW MYSQL 8.4 USES:
component_keyring_file

5. MOST IMPORTANT FILE:
/usr/sbin/mysqld.my

WITHOUT IT:
- component installs
- but encryption fails

6. IF RESTORE FAILS:
- use ENCRYPTION='N'
- restore first
- re-enable later

###############################
END
###############################

Kubernetes Networking Notes

by abhilashthale - June 3, 2026 coding

Kubernetes Networking Revision Notes

1. Basic Networking Building Blocks

Network Namespace

A Pod gets its own network namespace.

Inside a namespace:

Interfaces
Routing table
ARP table
iptables view

are isolated from the host.

Think:

Host Network Namespace ↓ Pod Network Namespace

veth Pair

A veth pair is like a virtual cable.

Example:

Pod eth0 ↔ veth123

One end is inside Pod.

One end is on host.

Traffic enters/leaves Pod through veth.

Bridge

A bridge works like a virtual switch.

Example:

PodA ↓ veth ↓ Bridge ↓ veth ↓ PodB

Bridge works at Layer 2.

2. Routing

Routing decides:

"Where should packet go next?"

Example:

DST=172.16.212.50

Kernel checks routing table.

Chooses interface.

Sends packet.

Routing happens after DNAT.

3. Netfilter Hooks

Hooks are checkpoints inside kernel.

PREROUTING

Packet just arrived.

Used mainly for:

DNAT
Service translation

INPUT

Packet destined for local host.

Examples:

sshd
kubelet

FORWARD

Packet passing through host.

Examples:

PodA → PodB
NodePort → Pod

OUTPUT

Packet generated locally.

Examples:

kubelet
curl from host

POSTROUTING

Packet leaving system.

Used mainly for:

SNAT
MASQUERADE

4. NAT

DNAT

Change destination.

Example:

10.96.0.25 ↓ 172.16.212.50

Used by:

Services
NodePort
Ingress traffic

SNAT

Change source.

Example:

172.16.212.50 ↓ 192.168.241.141

Used when returning traffic.

MASQUERADE

Dynamic SNAT.

Commonly used when Pods access outside world.

5. Conntrack

Conntrack tracks connections.

Stores:

SRC IP SRC PORT DST IP DST PORT Protocol

Example:

192.168.1.10:50000 ↓ 10.96.0.25:80

States:

NEW ESTABLISHED RELATED INVALID

Purpose:

Return traffic mapping
NAT tracking
Firewall efficiency

6. Kubernetes Services

A Service is NOT:

Process
Pod
Interface

Service is:

A virtual IP + kube-proxy rules.

Example:

rabbitmq-service

10.96.0.25

Flow:

Pod ↓ 10.96.0.25 ↓ kube-proxy ↓ DNAT ↓ RabbitMQ Pod

7. kube-proxy

Watches:

Services
Endpoints

from API Server.

Creates:

iptables rules

Main Job:

Service IP ↓ Pod IP

Example:

10.96.0.25 ↓ 172.16.212.50

8. CoreDNS

Purpose:

Name ↓ Service IP

Example:

rabbitmq ↓ 10.96.0.25

Pod resolv.conf:

nameserver 10.96.0.10

Flow:

Application ↓ DNS Query ↓ 10.96.0.10 ↓ kube-proxy ↓ CoreDNS Pod ↓ Returns Service IP

CoreDNS watches API Server.

Keeps service records in memory.

Example:

rabbitmq → 10.96.0.25

mysql → 10.96.0.50

9. Calico

Purpose:

Pod networking
Routing
Network Policy

Calico uses Linux networking.

Creates:

Routes
iptables policy rules

Traffic:

PodA ↓ Host Network ↓ FORWARD ↓ Calico Rules ↓ PodB

10. Network Policy

Network Policy controls:

Allow
Deny

pod communication.

Implemented by Calico.

Calico creates iptables rules.

Example:

Allow:

frontend ↓ backend

Deny:

all others

11. iptables Chains

Chain = Collection of Rules.

Example:

FORWARD chain

contains many rules.

Can call another chain.

Example:

FORWARD ↓ CALICO-FORWARD ↓ Policy Rules

Think:

Main Chain ↓ Sub Chain ↓ Rules

12. Pod to Pod Communication

PodA ↓ veth ↓ Host Routing ↓ Calico ↓ Worker2 ↓ PodB

13. Pod to Service Communication

PodA ↓ Service IP 10.96.0.25 ↓ kube-proxy DNAT ↓ PodB

14. DNS Resolution

Application

curl rabbitmq

↓

CoreDNS

↓

10.96.0.25

↓

Application connects

↓

kube-proxy

↓

RabbitMQ Pod

15. Ingress

Ingress Object:

Routing Rules.

Example:

myrabbitmqui.com ↓ rabbitmq-service

Ingress Object does NOT receive traffic.

16. Ingress Controller

Actual software.

Examples:

NGINX
Traefik

Receives HTTP/HTTPS traffic.

Reads:

Host Header

Example:

Host: myrabbitmqui.com

↓

rabbitmq-service

17. HTTPS

Browser ↓ 443 ↓ Ingress Controller ↓ Certificate Validation ↓ Encrypted Traffic

Certificate usually stored in:

Kubernetes Secret

18. Complete Flow

Browser ↓ myrabbitmqui.com ↓ DNS ↓ Worker IP / LoadBalancer ↓ Ingress Controller ↓ rabbitmq-service ↓ kube-proxy ↓ RabbitMQ Pod

Mental Model

CoreDNS

Name → Service IP

kube-proxy

Service IP → Pod IP

Calico

Can packet travel there?

Routing

How packet reaches destination

Ingress

Host Name → Service

Conntrack

Connection Tracking

DNAT

Change Destination

SNAT

Change Source

Service

Virtual IP

Pod

Actual Application

Linux To Kubernetes Networking - Complete Mental Model

by abhilashthale - June 3, 2026 coding

Linux to Kubernetes Networking - Complete Mental Model

The Most Important Principle

Kubernetes networking is not a separate networking stack.

Kubernetes uses Linux networking.

Everything eventually becomes:

Packet → Linux Kernel → Routing → Netfilter → Socket → Application

Kubernetes components such as CoreDNS, kube-proxy, Calico, Services, and Ingress only add rules and automation on top of Linux networking.

Part 1 - How a Packet Enters Linux

Suppose a packet arrives from the network cable.

The Network Interface Card (NIC) receives the Ethernet frame.

The NIC places the frame into its RX Ring Buffer. This buffer is memory that allows packets to be temporarily stored until the kernel can process them.

The NIC uses DMA (Direct Memory Access) to copy the packet directly into RAM without involving the CPU. This improves performance because the CPU does not need to manually move every packet.

After copying the packet to memory, the NIC notifies the kernel.

Modern Linux uses NAPI instead of processing every packet immediately. NAPI allows packets to be processed in batches, preventing interrupt storms during high traffic.

The kernel then creates an SKB (Socket Buffer).

The SKB is the most important structure in Linux networking.

An SKB contains:

Packet data
Source IP
Destination IP
Source Port
Destination Port
Protocol information
Metadata used by the kernel

Think of an SKB as the kernel's internal representation of a packet.

Part 2 - Ethernet Processing

The kernel first examines the Ethernet header.

The Ethernet header contains:

Source MAC Address
Destination MAC Address
EtherType

The EtherType tells Linux what protocol is inside the frame.

Examples:

0x0800 = IPv4 0x86DD = IPv6 0x0806 = ARP

If EtherType indicates IPv4, Linux passes the packet to the IP layer.

Part 3 - IP Layer

At the IP layer Linux examines:

Source IP Destination IP

Example:

Source IP = 192.168.1.10 Destination IP = 192.168.241.141

Before Linux decides where to send the packet, Netfilter hooks are executed.

Part 4 - Netfilter Hooks

Netfilter is a framework inside the Linux kernel.

It allows firewall, NAT and packet modification logic.

The five major hooks are:

PREROUTING INPUT FORWARD OUTPUT POSTROUTING

PREROUTING

This hook runs before Linux makes a routing decision.

Typical uses:

DNAT
Service Translation
NodePort Translation

Example:

Packet arrives:

Destination = 10.96.0.25

kube-proxy may change:

10.96.0.25

to

172.16.212.50

before routing happens.

INPUT

INPUT is used when the packet is meant for the local machine itself.

Examples:

SSH Destination Port 22

Kubelet Destination Port 10250

The packet eventually reaches a socket owned by a process running on the host.

FORWARD

FORWARD is used when the packet is not for the host.

The host acts like a router.

Examples:

PodA → PodB

NodePort → Pod

Ingress Controller → Application Pod

Most Kubernetes traffic goes through FORWARD.

OUTPUT

Used for packets created by the host itself.

Examples:

curl from the node

kubelet talking to API Server

POSTROUTING

Executed after routing decision and immediately before the packet leaves the system.

Used mainly for:

SNAT MASQUERADE

Part 5 - Routing

Routing determines where a packet should go.

Linux checks:

Destination IP

and consults:

Routing Table

The routing table tells Linux:

Which interface should carry the packet.

Which next hop should be used.

Routing happens after PREROUTING.

This is why DNAT must happen before routing.

If the destination IP changes, Linux needs the new destination before choosing the route.

Part 6 - NAT

NAT means modifying addresses.

There are three major types.

DNAT

Destination NAT changes the destination address.

Example:

Destination = 10.96.0.25

becomes

Destination = 172.16.212.50

Used by:

Services NodePort Ingress

SNAT

Source NAT changes the source address.

Example:

Source = 172.16.212.50

becomes

Source = 192.168.241.141

Used when traffic leaves the node.

MASQUERADE

MASQUERADE is dynamic SNAT.

Used when the external IP can change.

Commonly used when Pods access the Internet.

Part 7 - Conntrack

Conntrack is the kernel's connection tracking database.

It remembers:

Source IP Source Port Destination IP Destination Port Protocol

Suppose:

172.16.212.50:50000

connects to

10.96.0.25:80

After DNAT:

10.96.0.25

becomes

172.16.212.10

Conntrack remembers this translation.

When the response returns, conntrack ensures the reverse translation happens correctly.

Without conntrack, NAT would not work reliably.

Part 8 - Kubernetes Pod Networking

Every Pod gets its own Network Namespace.

Inside the namespace the Pod has:

Its own interfaces
Its own routing table
Its own ARP table

To connect the Pod to the host, Linux creates a veth pair.

Think of a veth pair as a virtual Ethernet cable.

One side exists inside the Pod.

The other side exists on the host.

Traffic leaving the Pod travels through this virtual cable.

Part 9 - Services

A Service is not an application.

A Service is not a network interface.

A Service is not a process.

A Service is a virtual IP managed by kube-proxy.

Example:

rabbitmq-service

ClusterIP:

10.96.0.25

No application actually listens on 10.96.0.25.

kube-proxy intercepts packets destined for 10.96.0.25 and translates them to a real Pod IP.

Part 10 - kube-proxy

kube-proxy watches the API Server.

It learns:

Services Endpoints

It then creates iptables rules.

Example:

10.96.0.25

becomes

172.16.212.50

This translation is performed using DNAT.

Therefore the primary job of kube-proxy is:

Service IP → Pod IP

Part 11 - CoreDNS

Applications prefer names rather than IP addresses.

Example:

rabbitmq

instead of

10.96.0.25

CoreDNS watches the API Server and maintains DNS records in memory.

Example:

rabbitmq

→

10.96.0.25

When a Pod performs DNS lookup, CoreDNS returns the Service IP.

CoreDNS performs:

Name → Service IP

kube-proxy performs:

Service IP → Pod IP

Part 12 - Calico

Calico provides Pod networking and Network Policies.

Calico does not forward packets itself.

Linux performs the forwarding.

Calico programs:

Routes iptables rules Network Policies

Think:

Linux does the work.

Calico tells Linux what rules to use.

Part 13 - Ingress

An Ingress object is only a routing rule.

Example:

myrabbitmqui.com

→

rabbitmq-service

The Ingress object itself receives no traffic.

An Ingress Controller such as NGINX reads the Ingress object and performs the routing.

NGINX receives the request, examines the Host header, and chooses the correct Service.

Complete Browser to RabbitMQ Flow

Browser opens:

https://myrabbitmqui.com

DNS returns:

192.168.241.141

Packet arrives at Worker Node.

NIC receives frame.

DMA copies frame into RAM.

NAPI schedules packet processing.

Kernel creates SKB.

Ethernet layer processes frame.

IP layer processes packet.

PREROUTING executes.

kube-proxy translates WorkerIP:443 to Ingress Controller Pod IP.

Routing decision occurs.

Packet traverses FORWARD chain.

Calico policy is checked.

Packet reaches NGINX Ingress Controller.

NGINX reads:

Host: myrabbitmqui.com

NGINX chooses:

rabbitmq-service

NGINX sends a new request to:

10.96.0.25

PREROUTING executes again.

kube-proxy translates:

10.96.0.25

to

172.16.212.50

Routing occurs.

Packet traverses FORWARD chain.

Packet reaches RabbitMQ Pod.

TCP delivers the packet to RabbitMQ's socket.

RabbitMQ application processes the request and sends the response back.

Kubernetes Pod Mastery - Module 2

by abhilashthale - Aug. 2, 2026 coding

# Kubernetes Pod Mastery - Module 2 Notes
## Process Lifecycle, Restart Policies, Command & Args

------------------------------------------------------------
1. CONTAINER PROCESS LIFECYCLE
------------------------------------------------------------

Every container has ONE main process (PID 1).

The lifecycle of the container depends on this process.

• If the main process is running
→ Container = Running

• If the main process exits with Exit Code 0
→ Container = Completed

• If the main process exits with a non-zero Exit Code
→ Container = Error

Examples:

sleep 300
→ Running

echo "Hello Kubernetes"
→ Completed (Exit Code 0)

false
→ Error (Exit Code 1)

------------------------------------------------------------
2. RESTART POLICIES
------------------------------------------------------------

The kubelet monitors the main process of every container.

When the process exits, kubelet checks the Pod's restartPolicy before deciding whether to create a new container.

There are three restart policies.

A) restartPolicy: Never

• Never restart the container.

Exit Code 0
→ Completed

Exit Code 1
→ Error

------------------------------------------------------------

B) restartPolicy: OnFailure

• Restart only when the process exits with a non-zero Exit Code.

Exit Code 0
→ No Restart

Exit Code 1
→ Restart

------------------------------------------------------------

C) restartPolicy: Always

• Restart the container regardless of whether it exits successfully or fails.

Exit Code 0
→ Restart

Exit Code 1
→ Restart

------------------------------------------------------------
3. CrashLoopBackOff
------------------------------------------------------------

CrashLoopBackOff is NOT a restart policy.

It is a waiting state where kubelet repeatedly restarts a container and applies an exponential backoff delay before trying again.

CrashLoopBackOff can occur when:

• Application exits with Exit Code 1 repeatedly.

OR

• Application exits successfully (Exit Code 0) very quickly (for example sleep 2) while restartPolicy is Always.

The kubelet gradually increases the delay between restart attempts.

------------------------------------------------------------
4. COMMAND AND ARGS
------------------------------------------------------------

Kubernetes uses:

command
→ Executable

args
→ Arguments passed to the executable

Final execution is conceptually:

command[] + args[]

Example:

command:
- echo
- hello

args:
- kubernetes

Final execution:

echo hello kubernetes

The first element of command is the executable.

The remaining elements of command become arguments.

The args array is appended after those arguments.

------------------------------------------------------------
5. Docker ENTRYPOINT and CMD
------------------------------------------------------------

Docker images contain default startup information.

Docker ENTRYPOINT
=
Kubernetes command

Docker CMD
=
Kubernetes args

Images also store metadata like:

• ENTRYPOINT
• CMD
• ENV
• USER
• WORKDIR
• EXPOSE
• LABEL
etc.

------------------------------------------------------------
6. OVERRIDE RULES
------------------------------------------------------------

Case 1

No command
No args

Result:

Use Image ENTRYPOINT
+
Use Image CMD

------------------------------------------------------------

Case 2

Only args

Result:

Use Image ENTRYPOINT

Replace Image CMD with Pod args

------------------------------------------------------------

Case 3

Only command

Result:

Use Pod command

Ignore Image ENTRYPOINT

Ignore Image CMD

------------------------------------------------------------

Case 4

command + args

Result:

Use Pod command

Use Pod args

Ignore Image ENTRYPOINT

Ignore Image CMD

------------------------------------------------------------
7. HOW KUBERNETES STARTS A CONTAINER
------------------------------------------------------------

------------------------------------------------------------
8. INTERNAL FLOW
------------------------------------------------------------

The image is NEVER modified.

The image already contains metadata:

• ENTRYPOINT
• CMD
• ENV
• USER
• WORKDIR

Containerd pulls the image and reads this metadata.

Kubelet reads the Pod specification.

Kubelet sends any overrides such as:

• command
• args
• environment variables
• volume mounts
• security settings

Containerd combines:

Image defaults
+
Pod overrides

Containerd generates the OCI Runtime Specification.

Finally, runc executes the Linux process using execve().

------------------------------------------------------------
9. IMPORTANT CKA POINTS
------------------------------------------------------------

✓ Every container has one main process.

✓ Exit Code 0 = Completed.

✓ Exit Code non-zero = Error.

✓ Kubelet decides whether to restart a container.

✓ CrashLoopBackOff is a restart backoff state, not a restart policy.

✓ command specifies the executable.

✓ args specifies arguments.

✓ command overrides the image ENTRYPOINT.

✓ args overrides the image CMD.

✓ Images are read-only; Kubernetes never modifies an image.

✓ Containerd reads image metadata and combines it with kubelet's Pod configuration before starting the container.

------------------------------------------------------------
10. MY MENTAL MODEL
------------------------------------------------------------

Whenever I troubleshoot a Pod, I ask these questions:

1. What is the main process?

2. Is the process still running?

3. If it exited:
- What is the Exit Code?

4. What is the restartPolicy?

5. Is kubelet restarting the container?

6. Is the container in CrashLoopBackOff?

7. What executable (command) is Kubernetes starting?

8. What arguments (args) are being passed?

If I can answer these questions, I can usually determine why a Pod is in Running, Completed, Error, or CrashLoopBackOff.

Health Probes (Revision Notes)

by abhilashthale - Aug. 7, 2026 coding

==========================================================
MODULE 4 - HEALTH PROBES (REVISION NOTES)
==========================================================

##########################################################
1. WHY DO WE NEED PROBES?
##########################################################

A container can be:

1. Starting
2. Running but not ready for traffic
3. Running and healthy
4. Running but hung/dead internally

Kubelet cannot determine all of these states by only checking if
the process exists.

Therefore Kubernetes provides three probes:

1. Startup Probe
2. Readiness Probe
3. Liveness Probe

----------------------------------------------------------

##########################################################
2. STARTUP PROBE
##########################################################

Purpose:
--------
Checks whether an application can successfully complete startup.

Question it answers:
--------------------
"Can this application finish starting?"

Used for:
---------
- Slow starting applications
- Spring Boot
- Java applications
- Applications loading cache
- Database initialization

How it works:
-------------
Container Starts
↓
Startup Probe runs
↓
If Fail
↓
Keep checking
↓
If Success (ONLY ONCE)
↓
Startup Probe is DISABLED forever
↓
Readiness and Liveness begin

Important:
----------
• Runs only during startup.
• Runs until first success.
• After success it never runs again for that container.
• If container restarts, Startup Probe starts again because it is a NEW container instance.

Configuration:
--------------
initialDelaySeconds
periodSeconds
failureThreshold
timeoutSeconds

Maximum startup time allowed:

initialDelaySeconds +
(periodSeconds × failureThreshold)

If startup never succeeds within this time:
→ Kubelet restarts container.

----------------------------------------------------------

##########################################################
3. READINESS PROBE
##########################################################

Purpose:
--------
Determines whether a Pod should receive traffic.

Question it answers:
--------------------
"Can I serve user requests?"

If Readiness succeeds:

Ready=True
↓
EndpointSlice Controller adds Pod IP
↓
kube-proxy updates iptables/IPVS
↓
Service sends traffic

If Readiness fails:

Ready=False
↓
EndpointSlice Controller removes Pod IP
↓
kube-proxy updates iptables/IPVS
↓
Service stops sending traffic

IMPORTANT:
----------
Readiness NEVER restarts a container.

Container continues running.

Only traffic is stopped.

----------------------------------------------------------

##########################################################
4. LIVENESS PROBE
##########################################################

Purpose:
--------
Determines whether the application is still healthy.

Question it answers:
--------------------
"Am I still alive?"

If Liveness succeeds:

Nothing happens.

Application keeps running.

If Liveness fails:

FailureThreshold reached
↓
Kubelet kills container
↓
RestartPolicy checked
↓
Container restarted

Repeated failures:

Restart
↓
Restart
↓
Restart
↓
Exponential Backoff
↓
CrashLoopBackOff

IMPORTANT:
----------
Liveness DOES restart containers.

----------------------------------------------------------

##########################################################
5. EXECUTION ORDER
##########################################################

Container Starts
│
▼
Startup Probe
│
▼
Startup Success
│
├─────────────┐
▼ ▼
Readiness Liveness
│ │
▼ ▼
Traffic Restart if unhealthy

----------------------------------------------------------

##########################################################
6. KUBELET RESPONSIBILITIES
##########################################################

Kubelet:

• Runs Startup Probe
• Runs Readiness Probe
• Runs Liveness Probe
• Updates Pod Ready condition
• Restarts failed containers
• Applies Restart Policy

----------------------------------------------------------

##########################################################
7. STARTUP vs READINESS vs LIVENESS
##########################################################

Startup Probe

Purpose:
Can application finish startup?

Controls:
Startup phase only

Restarts:
YES (if startup never succeeds)

Traffic:
NO

----------------------------------------------------------

Readiness Probe

Purpose:
Can application receive traffic?

Controls:
Traffic routing

Restarts:
NO

Traffic:
YES

----------------------------------------------------------

Liveness Probe

Purpose:
Is application still healthy?

Controls:
Container restart

Restarts:
YES

Traffic:
NO

----------------------------------------------------------

##########################################################
8. STARTUP PROBE vs LIVENESS initialDelaySeconds
##########################################################

Liveness initialDelaySeconds

Meaning:

"Wait X seconds.
After that I expect application to be healthy."

Problem:

If startup takes longer than expected,
container gets killed repeatedly.

----------------------------------------------------------

Startup Probe

Meaning:

"I don't know how long startup takes.
I'll keep checking until startup succeeds
or maximum startup time expires."

This removes the need to guess startup time.

----------------------------------------------------------

##########################################################
9. CRASHLOOPBACKOFF
##########################################################

CrashLoopBackOff is NOT a Pod phase.

It is kubectl STATUS shown when:

Container repeatedly exits
AND
RestartPolicy causes restart
AND
Kubelet applies exponential restart delay.

Typical delays:

Restart immediately
↓
10 sec
↓
20 sec
↓
40 sec
↓
80 sec
↓
160 sec
↓
300 sec (maximum backoff)

After maximum delay,
Kubelet still continues retrying approximately every 300 seconds
until the Pod is deleted or the problem is fixed.

----------------------------------------------------------

##########################################################
10. PROBE TIMELINE
##########################################################

Container Starts
│
▼
Startup Probe
│
├── Fail → Keep waiting
│
└── Success
│
▼
Startup Probe Disabled
│
▼
Readiness Starts
│
▼
Pod Ready=True
│
▼
Service sends traffic
│
▼
Application hangs
│
├───────────────┐
▼ ▼
Readiness Fail Liveness Fail
│ │
▼ ▼
Stop Traffic Restart Container

----------------------------------------------------------

##########################################################
11. IMPORTANT CKA POINTS
##########################################################

✓ Startup runs only until first success.

✓ Startup runs again after every container restart.

✓ Readiness controls traffic only.

✓ Readiness NEVER restarts containers.

✓ Liveness restarts unhealthy containers.

✓ Startup disables Readiness and Liveness until startup succeeds.

✓ EndpointSlice Controller reacts to Ready=True/False.

✓ kube-proxy updates iptables/IPVS after EndpointSlice changes.

✓ Services only send traffic to Ready Pods.

✓ CrashLoopBackOff is caused by repeated restarts with exponential backoff.

==========================================================
END OF MODULE 4
==========================================================

===========================================================
CKA POD MASTERY
MODULE 4 – HEALTH PROBES (MASTER NOTES)
===========================================================

############################################################
WHY KUBERNETES NEEDS PROBES
############################################################

Kubelet can only see:

Container Process
Running?
Exit Code?

It CANNOT know:

✓ Is application still loading?
✓ Is application ready?
✓ Is application hung?
✓ Can application serve requests?

Therefore Kubernetes provides three probes.

------------------------------------------------------------

STARTUP PROBE

Question:
"Can application finish startup?"

READINESS PROBE

Question:
"Can application receive traffic?"

LIVENESS PROBE

Question:
"Is application still healthy?"

------------------------------------------------------------

Think like this:

Startup
↓

Readiness
↓

Liveness

============================================================

STARTUP PROBE
============================================================

Purpose

Protect slow-starting applications.

Examples

Spring Boot
Java Applications
Large Cache Loading
Database Migration
Machine Learning Models

Flow

Container Starts

↓

Startup Probe Runs

↓

Fail

↓

Keep Waiting

↓

Fail

↓

Keep Waiting

↓

Success

↓

Disable Startup Probe Forever

↓

Enable Readiness

↓

Enable Liveness

IMPORTANT

Startup Probe runs ONLY during startup.

Once it succeeds once,
it NEVER runs again
for THAT container.

If container restarts,
Startup Probe starts again.

WHY?

Because probes belong to
container instance.

Container #1

↓

Startup Passed

↓

Container Dies

↓

Container #2

↓

Startup runs AGAIN

============================================================

REAL EXAMPLE
============================================================

Application Startup

sleep 30

touch /tmp/startup-ok

touch /tmp/ready

sleep 300

Startup checks

cat /tmp/startup-ok

Timeline

0 sec

Container Starts

↓

Startup Probe

↓

Fail

↓

Fail

↓

touch /tmp/startup-ok

↓

Startup Success

↓

Startup Disabled

↓

Readiness Starts

↓

Liveness Starts

============================================================

READINESS PROBE
============================================================

Purpose

Should this Pod receive traffic?

If Ready=True

↓

API Server updated

↓

EndpointSlice Controller

↓

Pod IP Added

↓

kube-proxy

↓

iptables/IPVS updated

↓

Traffic Starts

-------------------------------------

If Ready=False

↓

API Server updated

↓

EndpointSlice Controller

↓

Remove Pod IP

↓

kube-proxy updates

↓

NO TRAFFIC

IMPORTANT

Readiness NEVER

kills

restarts

or recreates container.

It ONLY controls traffic.

============================================================

REAL EXAMPLE
============================================================

Application Running

↓

rm /tmp/ready

↓

Readiness Probe

↓

Fail

↓

Ready=False

↓

EndpointSlice removes Pod

↓

Service Stops Sending Traffic

Container STILL RUNNING

============================================================

LIVENESS PROBE
============================================================

Purpose

Is application still alive?

If Healthy

↓

Nothing

If Unhealthy

↓

Failure Threshold Reached

↓

Kubelet

↓

Kill Container

↓

Restart Policy

↓

Restart Container

============================================================

REAL EXAMPLE
============================================================

Application

touch /tmp/ready

↓

Later

rm /tmp/ready

↓

Liveness

Fail

↓

Fail

↓

Fail

↓

Kubelet Restarts Container

============================================================

STARTUP vs READINESS vs LIVENESS
============================================================

Startup

Protect Startup

Runs once

Can Restart

No Traffic Control

-------------------------------------

Readiness

Traffic Control

Runs continuously

Never Restarts

Controls EndpointSlice

-------------------------------------

Liveness

Health Check

Runs continuously

Restarts Container

No Traffic Control

============================================================

INITIAL DELAY
============================================================

Startup initialDelay

Wait before checking startup.

Useful when app
ALWAYS needs
some minimum time.

------------------------------------------------

Liveness initialDelay

Wait before checking health.

Problem

You must GUESS startup time.

If app starts slower

↓

Liveness kills container.

This is WHY Startup Probe exists.

============================================================

FAILURE THRESHOLD
============================================================

Example

periodSeconds = 10

failureThreshold = 3

Three consecutive failures

↓

Action

Readiness

↓

NotReady

Liveness

↓

Restart

Startup

↓

Startup Failed

Restart Container

============================================================

CRASHLOOPBACKOFF
============================================================

CrashLoopBackOff

means

Container keeps restarting.

Kubelet applies

Exponential Delay

Typical

Restart

↓

10 sec

↓

20 sec

↓

40 sec

↓

80 sec

↓

160 sec

↓

300 sec

↓

300 sec

↓

300 sec

continues forever

until fixed.

============================================================

COMPLETE TIMELINE
============================================================

Container Starts

↓

Startup Probe

↓

Success

↓

Readiness

↓

Ready=True

↓

Service sends traffic

↓

Application hangs

↓

Readiness fails

↓

Traffic Stops

↓

Liveness fails

↓

Restart

↓

NEW CONTAINER

↓

Startup Probe AGAIN

============================================================

MY MISTAKES (VERY IMPORTANT)
============================================================

Mistake 1

I wrote

command:

- sleep

- 30

- touch /tmp/file

Wrong

Reason

command is NOT shell script.

First value

Executable

Remaining values

Arguments

Correct

command:

- sh

- -c

- |
sleep 30
touch /tmp/file

------------------------------------------------

Mistake 2

I thought

CrashLoopBackOff

means Pod stopped.

Reality

Pod can show

Running

while kubelet
is retrying container.

CrashLoopBackOff

is restart state,
NOT pod phase.

------------------------------------------------

Mistake 3

I thought

Readiness restart container.

Reality

Readiness NEVER restarts.

Only removes traffic.

------------------------------------------------

Mistake 4

I thought

Startup never runs again.

Reality

Startup runs again

AFTER EVERY CONTAINER RESTART.

------------------------------------------------

Mistake 5

Confused with seconds.

Solution

STOP calculating time.

Instead

Draw Timeline.

Example

Container

↓

Startup

↓

Ready

↓

Traffic

↓

Readiness Fail

↓

Traffic Stops

↓

Liveness Fail

↓

Restart

Much easier than counting seconds.

============================================================

CKA EXAM POINTS
============================================================

✓ Startup protects startup.

✓ Readiness controls traffic.

✓ Liveness restarts containers.

✓ Startup disables
Readiness & Liveness
until startup succeeds.

✓ Readiness modifies
EndpointSlice.

✓ kube-proxy updates
iptables/IPVS.

✓ Services send traffic
ONLY to Ready Pods.

✓ Startup belongs to
container instance.

✓ CrashLoopBackOff
uses exponential backoff.

============================================================
END OF MODULE 4
============================================================

Pod Mastery

by abhilashthale - Aug. 15, 2026 coding

POD MASTERY
│
├── 1. Pod Fundamentals
│ ├── Pod Concept
│ ├── Smallest Deployable Unit
│ ├── Pod vs Container
│ ├── Multi-Container Pod
│ ├── Shared Network Namespace
│ ├── Shared Storage
│ └── Shared Pod Resources
│
├── 2. Pod Lifecycle
│ ├── Pod Phases
│ │ ├── Pending
│ │ ├── Running
│ │ ├── Succeeded
│ │ ├── Failed
│ │ └── Unknown
│ │
│ ├── Container States
│ │ ├── Waiting
│ │ ├── Running
│ │ └── Terminated
│ │
│ ├── Restart Policy
│ │ ├── Always
│ │ ├── OnFailure
│ │ └── Never
│ │
│ ├── Kubelet Container Monitoring
│ └── Container Exit Status
│
├── 3. Pod Initialization & Containers
│ ├── Init Containers
│ ├── Init Container Ordering
│ ├── Init Container Success/Failure
│ ├── Multiple Init Containers
│ ├── Multi-Container Pods
│ ├── Container Communication
│ └── Sidecar Concept
│
├── 4. Pod Networking
│ ├── Pod Network Namespace
│ ├── Pod IP
│ ├── Container-to-Container Communication
│ ├── localhost Inside Pod
│ ├── Network Namespace Sharing
│ ├── veth Pair
│ ├── CNI
│ └── Pod Network Flow
│
├── 5. Pod Storage
│ ├── Volume Concept
│ ├── Volume Lifecycle
│ │
│ ├── emptyDir
│ │ ├── Pod Lifetime
│ │ ├── Container Restart
│ │ └── Shared Between Containers
│ │
│ ├── hostPath
│ │ ├── Node Filesystem
│ │ ├── Node Dependency
│ │ └── DirectoryOrCreate
│ │
│ ├── ConfigMap Volume
│ ├── Secret Volume
│ │ ├── data
│ │ ├── stringData
│ │ └── Base64 Encoding
│ │
│ ├── Projected Volume
│ │ ├── ConfigMap
│ │ ├── Secret
│ │ ├── Downward API
│ │ └── ServiceAccount Token
│ │
│ └── Downward API
│ ├── Pod Metadata
│ ├── Pod Name
│ ├── Namespace
│ └── Pod Fields
│
├── 6. Pod Resources
│ ├── Resource Requests
│ ├── Resource Limits
│ ├── CPU
│ ├── Memory
│ ├── Ephemeral Storage
│ ├── Scheduler Uses Requests
│ ├── Limits Enforcement
│ ├── QoS Classes
│ │ ├── Guaranteed
│ │ ├── Burstable
│ │ └── BestEffort
│ ├── OOMKilled
│ └── Resource Pressure
│
├── 7. Pod Health & Probes
│ ├── Why Probes Are Needed
│ ├── Liveness Probe
│ ├── Readiness Probe
│ ├── Startup Probe
│ ├── Probe Types
│ │ ├── HTTP
│ │ ├── TCP
│ │ ├── Exec
│ │ └── GRPC
│ ├── Probe Timing
│ │ ├── initialDelaySeconds
│ │ ├── periodSeconds
│ │ ├── timeoutSeconds
│ │ ├── failureThreshold
│ │ └── successThreshold
│ └── Probe Failure Behavior
│
├── 8. Pod Identity & RBAC
│ ├── ServiceAccount
│ ├── Default ServiceAccount
│ ├── serviceAccountName
│ ├── ServiceAccount Token
│ ├── automountServiceAccountToken
│ ├── Authentication
│ ├── Authorization
│ ├── Role
│ ├── ClusterRole
│ ├── RoleBinding
│ ├── ClusterRoleBinding
│ ├── Namespace Scope
│ └── Cluster-Wide Scope
│
├── 9. Pod SecurityContext
│ ├── Why SecurityContext
│ ├── Pod-Level vs Container-Level
│ ├── runAsUser
│ ├── runAsGroup
│ ├── supplementalGroups
│ ├── fsGroup
│ ├── runAsNonRoot
│ ├── readOnlyRootFilesystem
│ ├── Linux Capabilities
│ │ ├── add
│ │ └── drop
│ ├── privileged
│ ├── allowPrivilegeEscalation
│ └── seccompProfile
│ ├── RuntimeDefault
│ ├── Unconfined
│ └── Localhost
│
├── 10. Pod Security Standards
│ ├── PSS
│ │ ├── Privileged
│ │ ├── Baseline
│ │ └── Restricted
│ │
│ ├── Pod Security Admission
│ ├── Admission Phase
│ ├── Namespace Labels
│ ├── enforce
│ ├── warn
│ └── audit
│
├── 11. Pod Scheduling
│ ├── Scheduler Flow
│ ├── nodeName
│ ├── nodeSelector
│ ├── Node Affinity
│ ├── Pod Affinity
│ ├── Pod Anti-Affinity
│ ├── topologyKey
│ ├── Taints
│ │ ├── NoSchedule
│ │ ├── PreferNoSchedule
│ │ └── NoExecute
│ └── Tolerations
│ ├── Equal
│ └── Exists
│
├── 12. Pod Termination
│ ├── Pod Deletion
│ ├── Termination Grace Period
│ ├── preStop Hook
│ ├── SIGTERM
│ ├── SIGKILL
│ └── Graceful Shutdown
│
├── 13. Pod State & Conditions
│ ├── Pod Phase
│ ├── Pod Conditions
│ │ ├── PodScheduled
│ │ ├── Initialized
│ │ ├── ContainersReady
│ │ └── Ready
│ ├── Container State
│ └── Phase vs Condition vs Container State
│
├── 14. Pod DNS
│ ├── Pod Hostname
│ ├── /etc/hosts
│ ├── /etc/resolv.conf
│ ├── CoreDNS
│ ├── DNS Search Domains
│ ├── ndots
│ ├── dnsPolicy
│ └── hostNetwork Interaction
│
├── 15. Pod Eviction & Node Pressure
│ ├── MemoryPressure
│ ├── DiskPressure
│ ├── PIDPressure
│ ├── Ephemeral Storage Pressure
│ ├── Kubelet Eviction
│ ├── QoS and Eviction
│ └── OOMKilled vs Eviction
│
├── 16. Static Pods
│ ├── Static Pod Concept
│ ├── Kubelet Managed
│ ├── Static Pod Manifest
│ ├── /etc/kubernetes/manifests/
│ ├── Static Pod vs Normal Pod
│ └── Control Plane Static Pods
│
└── 17. Pod Troubleshooting & Final Mastery
├── Pending
├── CrashLoopBackOff
├── ImagePullBackOff
├── CreateContainerConfigError
├── ContainerCreating
├── OOMKilled
├── Permission Denied
├── Volume Mount Failure
├── ConfigMap/Secret Failure
├── Probe Failure
├── SecurityContext Failure
├── DNS Failure
├── Scheduling Failure
├── Node Pressure / Eviction
└── Final Pod Mastery Lab & Test

Calico Pod Communication

by abhilashthale - Aug. 21, 2026 coding

## 🧱 1. The Core Infrastructure Components

Before any traffic moves, Calico and Kubernetes set up the landscape:

* The veth Pairs: For every Pod created, Calico creates a virtual ethernet cable. One end sits in the Pod's network namespace; the other end sits in the host namespace (named caliXXXXX).

* The Map (Routing Table): Calico’s agent (Felix) pre-populates the Linux kernel’s native routing table (ip route). It writes rules mapping every Pod IP to its specific caliXXXXX interface.

* The Security Guard (Netfilter/iptables): Calico and kube-proxy inject custom sub-chains (starting with cali- and KUBE-) directly into the kernel's native Netfilter hooks.

------------------------------

## 🕒 2. The Strict Order of Operations

When a packet travels from Pod 1 to a Service VIP, it triggers Netfilter hooks and routing decisions in a strict, unchangeable timeline:

## Hook 1: PREROUTING (The Entry Checkpoint)

* kube-proxy hits first: It intercepts the virtual Service VIP and executes DNAT, rewriting the destination address to a Real Backend Pod IP.

* Calico hits immediately after: It runs its cali-PREROUTING chain. It inspects the packet after DNAT and applies Ingress Network Policies.

* Action: It either permits the packet or drops it instantly to save CPU cycles.

## Step 2: The Routing Decision (The Intersection)

* The Linux Kernel takes over: The kernel looks at the packet's destination (the Real Pod IP).

* The Map Lookup: It references the routing table that Calico pre-configured.

* Action: The kernel matches the Pod IP and determines the exact exit interface (caliXXXXX for same-host, or a physical NIC like eth0 for cross-host).

## Hook 3: FORWARD (The Transit Lane)

* The Context: Because the packet is crossing from one network interface namespace to another, the kernel routes it through the FORWARD hook.

* Calico hits here: It runs its cali-FORWARD chain to validate transit safety.

* Action: It double-checks that Pod 1 is allowed to talk to Pod 2. If a policy blocks it, the packet is killed here.

## Hook 4: POSTROUTING (The Exit Gate)

* The Context: The packet has been routed and cleared by security. It is sitting at the exit interface, ready to leave.

* Calico/kube-proxy hit here: They run the cali-POSTROUTING chain to enforce Egress Network Policies.

* Action: If the packet is exiting the cluster to the external internet, SNAT (Masquerading) is applied at the very last second, rewriting the source IP to the Node's IP so replies can find their way back.

------------------------------

## 🎯 3. Visual Execution Flow

[ Packet Leaves Pod 1 via veth ] ──► Enters Host Namespace

│

▼

┌─────────────────────────────────────────────────────────────────────────────────┐

│ KERNEL HOOKS & CALICO INTERACTION │

│ │

│ 1. PREROUTING HOOK │

│ ├──► kube-proxy: DNAT (VIP ──► Real Pod 2 IP) │

│ └──► Calico: Checks Ingress Policy ──► [ALLOW / DROP] │

│ │

│ 2. ROUTING DECISION │

│ └──► Linux Kernel: Reads Calico's pre-written routing table │

│ └──► Destination matched ──► Target Interface locked in │

│ │

│ 3. FORWARD HOOK │

│ └──► Calico: Checks Transit/Inter-pod Policy ──► [ALLOW / DROP] │

│ │

│ 4. POSTROUTING HOOK │

│ ├──► Calico: Checks Egress Policy ──► [ALLOW / DROP] │

│ └──► kube-proxy/Calico: Applies SNAT (only if traffic leaves node/cluster) │

└─────────────────────────────────────────────────────────────────────────────────1

│

▼

[ Packet Enters Target veth ] ──► Arrives safely at Pod 2!

------------------------------

## 💡 4. The Golden Rules to Remember

1. Linux Kernel does the driving: Calico never routes a packet mid-flight. The kernel does the routing based on a map Calico drew ahead of time.

2. Calico is the firewall: Calico's primary job inside the Netfilter hooks is executing immediate Allow/Deny decisions based on your Network Policies.

3. DNAT changes the target, Routing finds the path, SNAT changes the source: DNAT happens first so the kernel knows where to route the packet. SNAT happens last so it doesn't mess up the routing decision or Calico's policy evaluation. [1]

Now that you have the complete summary, where would you like to go next?

* We can look at how this changes if you switch Calico to VXLAN/IPIP Overlay mode instead of native routing.

* We can look at how Calico's eBPF mode deletes all of these iptables hooks entirely for faster networking.

Blog Posts

Rabbitmq Cluster On Kubernetes (Complete Setup + Troubleshooting Guide)

Kubernetes Imperative Vs Declarative

Mysql 8.0.X To 8.4.X Upgrade Runbook (Clean Install + Keyring Migration + Encryption Testing)

Kubernetes Networking Revision Notes

1. Basic Networking Building Blocks

Network Namespace

veth Pair

Bridge

2. Routing

3. Netfilter Hooks

PREROUTING

INPUT

FORWARD

OUTPUT

POSTROUTING

4. NAT

DNAT

SNAT

MASQUERADE

5. Conntrack

6. Kubernetes Services

7. kube-proxy

8. CoreDNS

9. Calico

10. Network Policy

11. iptables Chains

12. Pod to Pod Communication

13. Pod to Service Communication

14. DNS Resolution

15. Ingress

16. Ingress Controller

17. HTTPS

18. Complete Flow

Mental Model

CoreDNS

kube-proxy

Calico

Routing

Ingress

Conntrack

DNAT

SNAT

Service

Pod

Linux to Kubernetes Networking - Complete Mental Model

The Most Important Principle

Part 1 - How a Packet Enters Linux

Part 2 - Ethernet Processing

Part 3 - IP Layer

Part 4 - Netfilter Hooks

PREROUTING

INPUT

FORWARD

OUTPUT

POSTROUTING

Part 5 - Routing

Part 6 - NAT

DNAT

SNAT

MASQUERADE

Part 7 - Conntrack

Part 8 - Kubernetes Pod Networking

Part 9 - Services

Part 10 - kube-proxy

Part 11 - CoreDNS

Part 12 - Calico

Part 13 - Ingress

Complete Browser to RabbitMQ Flow

Kubernetes Pod Mastery - Module 2

Health Probes (Revision Notes)

Pod Mastery