Part2 - Ephemeral Pods in Kubernetes

8 min readNov 18, 2020

Smooth Transition to Kubernetes

Before continuing the post, I recommend you to check the news-tracker-application on Github and the first part of the series, Kubernetes Architecture. As of now, I’m frequently going to use the Kubernetes resources in that project. Let’s get started with another component of the Kubernetes.

What is a Pod?

If you want to run your app(s) on Kubernetes, then, need a POD which consists of the container(s). A pod is the smallest deployable unit of Kubernetes and consists of one or more containers, which are relatively tightly coupled. The Pods run together like a logic unit and are managed by the kube-proxy and kubelet running in the worker node. All Pods don’t need to run on the same machine as containers. They can span more than one machine, which means multiple pods can run on multiple nodes. All the containers in a pod have the same IP address, port space, network interface shared resources such as CPU, RAM, and Volume. They can communicate using localhost or standard inter-process communication. Let’s expand that communication thingy.

Inter-Pod Communication

Every pod can access every other pod at the other pod’s IP address. No NAT (Network Address Translation) gateways exist between them. This is usually achieved through an additional software-defined network layered on top of the actual network. Containers in one Pod run in the same network, which means sharing the same IP address and port space. That’s why containers in the same Pod should pay attention not to bind the same port which causes port conflict, check Pod 5; Container 4 and 5 ports in the figure above. On the other side containers in the same Pod can communicate with each other through “localhost” by means of a loopback network interface. Each container can also communicate with any other pod or service within the cluster. I want to go inside the pods to describe what those mean ;

Please before starting, check the Requirements for running Kubernetes locally and be sure a Minikube is properly working in your local.

Create and list the Pod(s) :

$ kubectl apply -f  https://raw.githubusercontent.com/ErhanCetin/k8s-smooth-transition/master/k8s/medium-post/pods/news-tracker-activemq-pod.yaml$ kubectl apply -f  https://github.com/ErhanCetin/k8s-smooth-transition/blob/master/k8s/medium-post/pods/news-tracker-job-pod.yaml$ kubectl apply -f  https://raw.githubusercontent.com/ErhanCetin/k8s-smooth-transition/master/k8s/medium-post/pods/news-tracker-api-pod.yaml$ kubectl get pods -o wide # list all pods.

Check — Different IP addresses are assigned to every pod when the pod(s) are created.

Accessing from one container to another:

→ Let’s access the newsapi-pod’s container in the activemq-pod’s container through IP.

Check the container port and IP of the newsapi-pod’s container :

$ kubectl describe pods newsapi-pod

Now, we know the IP (172.17.0.4) of the “newsapi-pod” and the port (8080) of the “newsapi” container. The time is to open a session in the “activemq” container and execute “curl” inside the container to access the “newsapi” container.

$ kubectl exec -it activemq-pod activemq -- /bin/bash

I would show accessing from pod to another in separate nodes as well. But, with Minikube installation, there is just one node and one cluster in our local. But you can be sure, any pod can access any other pod in different nodes, but the same cluster, via IP addresses. If you wish to test more than one node, you can use a kind solution to have more nodes in your local.

What is going on with Kubernetes when you create a Pod?

We’ve known so far how to create pods and also checked the communication between them. Now, it’s time to investigate what actually happens during a pod creation process. I’m going to explain a bit of detail, but please check Kubernetes Architecture to refresh your memory to catch the components which are used in the image (A Pod Creation Flow) below.
Side Note: In our Pod YAML file, a Pod resource is being directly used to create a pod. But, in real development, A Replication Controller or Deployment Resources are being used for the creation of Pods. I will explain the Replication and Deployment Controllers in the following post. Let’s roll our sleeves up :
First, when Kubernetes starts, every Kubernetes components register itself to API Server and starts to listener for the API Server actions. When you create a Pod via a Replication or Deployment Controller, those controllers compare the actual state and desired state of the Pod by using data which is sent by a kubelet running on worker node and decide whether the Pod(s) should be created or not. Let say, it is decided to create a Pod after comparing the actual and desired state of the Pod. In this case, the Scheduler gets notified. The Scheduler starts to compute data which is again sent by a Kubelet beforehand and decides a worker node so that the Pod(s) are deployed. After making a decision, result data is sent to API Server. This time, the Kubelet gets notified and it gets the Pod specification through API Server and creates actual Pod(s) on the worker node which it runs on.
I guess this long explanation is a bit hard to follow the order of components. Please check the diagram below from the Kubernetes in Action book. This helps you to understand the actual flow of creating the Pod.

Why do we need multiple containers in a Pod?

One of my favorite approaches in the software environment is “enables decoupling software dependencies”. When we look at it from a container perspective, we can’t say something against having multiple-container in the same Pod due to the “one process per container” principle. Having multiple containers in the same Pod is totally violates the principle. Isn’t it? Just think, you have multiple containers in one Pod and have some problem with one of the container, in this case, can’t it be harder to troubleshoot the pod? You have to check container logs for all process so that understand which process is doing what. And probably, you will spend much more time than checking just only one process log. Likewise, this also prevents us to utilise our infrastructure, means we will not able to re-schedule our resource like CPU and memory, destroying scaling… But, sorry for that ( I’m contradicting myself ), sometimes we actually need multiple containers in the same pod to support co-located and co-managed helper for a primary application-container. Container except main application is called as “Helper/SideCar” container. Actually, this is handled in “Multi-Container Design Pattern” like “Proxies, bridges, and adapters design pattern”. Those containers are used for various purposes such as Log Shippers, Data Loader, Monitoring Agents, and Proxies in Service Mesh Architecture ….

Let’s have a look at various topics about the Pod and finish this post.

How to check a Pod Log :

$ kubectl get pods -o wide
$ kubectl logs activemq-pod

Note: if we have multiple containers in a Pod, then :

$ kubectl get logs <pod name> <container-name>

One more thing, The logs are automatically rotated daily whenever the log file reaches 10MB in size. Also, the logs are deleted when the pod is deleted. Of course, we can prevent this by installing cluster-wide logging applications like the fluent.

Terminating a Pod :

If you don’t use a Deployment/Replication Controller resource to create a Pod, you can delete a Pod as follows. But if you use them you have to delete the Deployment/Replication Controller instead of the Pod itself.

$ kubectl delete pod <pod-name>
$ kubectl delete pod <pod-name> — grace-period=0 # or with force mode

In the case of using deployment resource, the command below delete automatically all pods which are corresponding to the your-deployment resource. But, if you delete directly a pod instead of deployment, The Kubernetes ( via a kubelet ) will automatically create the container again.

$ kubectl delete deployment <deployment-node># Another way to delete a container instead of a Pod :$ kubectl exec -it pod-name -c container-name — /bin/sh -c “kill 1”
$ kubectl exec -it activemq-pod -c activemq — /bin/sh -c “kill 1”

What happens when a Pod is deleted :

The Kubernetes receives instructing to terminate all the containers that are part of that pod.

2. The Pod status is being set and mark it as “Terminating” to prevent receiving the traffics.

3. The Kubernetes sends a “TERM” signal to the Pod to start to shut down the Pod (the 30s by default).

4. If you don’t delete a Pod with a grace option like “ — grace-period=0“, Kubernetes waits the grace period and operates a “SIGKILL” to the processes inside the Pod.

5. Finally, the related Pod is removed from the etc via kubelet → ApiServer → etcd.

Debugging and Testing a Pod

Most of the time, we need debugging applications within the containers in our local. For that, we can use Port-Forward to forward our local network to the containers. The following commands will forward your machine’s local port 8161 to port 8161 of activemq-pod.

$ kubectl describe pod activemq-pod # Check the PORT of activemq-pod

$ kubectl port-forward activemq-pod 8161:8161 # start port-porward process
# Also, you can forward different port like 8165:8161
# 8165 : your local port, 8161: the container port.

→ Go to a web browser in your local and hit for “http://localhost:8161. You will see:

What happens when you use a local browser :

Here are some useful commands to manage the Pods. See you in the next post.

Useful Pod commands :

# get specific pod
$ kubectl get pod activemq-pod -o wide.# get pod list
$ kubectl get pods# get pod detail 
$ kubectl describe pod activemq-pod# get pod details in json format.
$ kubectl get pods activemq-pod -o=jsonpath='{@}'.# get a Pod IP
$ kubectl get pods activemq-pod -o jsonpath --template={.status.podIP}
$ kubectl describe pod activemq-pod | grep IP # get a Pod name
$ kubectl get pods activemq-pod -o=jsonpath='{.metadata.name}'# get a container name
$ kubectl get pods activemq-pod -o jsonpath='{.spec.containers[*].name}'# sort the Pods by node name 
$ kubectl get pod -o wide --sort-by=.spec.nodeName # for debugging - connect the Pod in your local.
$ kubectl port-forward <pod-name>  <localPort:servicePort># delete a pod.
$ kubectl delete pod activemq-pod# get a shell to a running container
$ kubectl exec -it <pod-name> <container-name> -- /bin/bash--> further command