Kubernetes is a popular cluster and orchestrator for containerised applications. It is quite easy to create an application image, deploy it to the cluster and run as a container. But in many production scenarios a single application consists of many cooperating processes that should be executed as separate containers. And the cooperation needs communication.
This article tries to gather and describe (on a high level) a few different ways of direct communication between containers run in Kubernetes cluster. Some of them were applied successfully in one of our projects, while building a microservices-based application.
Levels of communication
Multi-container applications can use various topologies of their running containers. Therefore, we could distinguish the following two general levels of inter-container communication in a Kubernetes cluster:
- Multiple containers communicating within a single Kubernetes
- Containers from different pods communicating each other
Communication within a pod
A pod is an atomic unit in Kubernetes. It means it is the smallest part that can be deployed and managed in a cluster. A pod is a logical host for one or more containers and optional shared resources (e.g. volumes). A common scenario for multiple containers in a pod is one primary application container run together with auxiliary sidecar containers (e.g. used for logging, communication, etc.). Another type of specific containers is an initialization container that is executed once before the primary container starts in a pod.
A pod is always deployed on a single cluster node (i.e. machine) within one container runtime (e.g. Docker). In result, all containers in a pod have the same logical host and share the same IP address and inter-process communication (IPC) namespace. The picture 1 shows relationships between nodes, pods and containers in a Kubernetes cluster.
In most scenarios a pod contains only one container, but when there is a need to run multiple tightly coupled ones, a few communication options exist:
- Inter-process communication (IPC). Because containers run within a single pod and share the same logical host, we can use standard methods of communication between processes. It includes shared memory, pipes, message queues and other techniques if they are supported by the used container runtime and underlying operating system.
- Shared file system. Kubernetes offers a Volume entity that can be created within a pod and shared by all its containers. Its lifetime is bound to the pod – the volume is destroyed when the pod ceases to exist. Such a volume can be mounted as a part of the file system available for the containers, that can interchange information via writing and reading files. There are many various types of available volumes (e.g. Azure disk, Kubernetes config map), but the emptyDir type should be enough for most cases. It just defines a new empty directory when a pod is initialized on a cluster node.
- All containers run within a pod share the same unique cluster IP address and port space. When containers are assigned with separate port numbers, they can communicate with any network protocol using the localhost address. Of course, they must be aware of these port numbers, so this approach requires proper containers configuration and ports coordination among different containers.
Despite pods in Kubernetes having unique IP addresses, it is not recommended to access them directly from another pod using such addresses. A pod in Kubernetes always can be recreated and assigned with a new cluster IP address that would result in breaking the pod-to-pod communication. The correct approach is to use services.
Service resource in Kubernetes defines a logical set of pods and is used to expose an application run in these pods (see the picture 2). Matching pods are found using the selector specified in service and pods definitions. A service can be accessed using an IP address/DNS name and port number(s). The service’s IP address can be generated automatically or specified in a definition. When a service is used by another process, Kubernetes services controller chooses one of the service’s pods and redirects the request to it. The pod selection method is dependent on used service type and cluster configuration (proxy mode). It could be for example random selection, round-robin or load-balancing.
But how a process running in a cluster can know a service address and connect to? When a container wants to obtain a service IP address, it can use special environment variables that Kubernetes creates for each running pod for each service. These variables contain services’ IP addresses, port numbers and protocol names. Alternatively, we can use internal DNS service (kube-dns add-on) in a cluster and access Kubernetes services via their local unique names.
There are four types of Kubernetes services. We define the service’s type as a part of its definition, and it implies the way we can access the service.
- ClusterIP. It is a default service type that makes the service available only internally within a cluster, using a generated or assigned cluster IP address. Every running container in cluster can access such services using its address and port number.
- NodePort. This type makes a service available outside a cluster using nodes public IP addresses and the same node port number. The node port number can be assigned automatically by Kubernetes or specified manually in the service’s definition. By default, it must be in the range 30000 – 32767, but the range can be also configured on the cluster kube-proxy level. So, when a process (inside or outside a cluster) wants to use such a service, it must know at least one public IP address of the Kubernetes cluster nodes and the node port number exposed on each node. For containers inside the cluster it is still possible to access the service using internal cluster IP address and internal port number (or via local DNS service).
- LoadBalancer. This type is primarily intended to expose a service externally, but containers within a cluster can also use services via a load balancer that selects a pod that should handle a request. It is important to note that Kubernetes does not provide any built-in load balancer product. It must be installed and configured separately and then connected to a Kubernetes cluster. Fortunately, many of cloud providers support external load balancers for their Kubernetes services. For example, Azure Kubernetes Service creates a load balancer for a cluster automatically, that is ready for use with services of LoadBalancer Load-balanced services can be accessed using an external public IP address of the load balancer and sometimes this address can be set in the service’s definition. Of course, services of LoadBalancer type can still be visible internally via cluster IP address.
- ExternalName. This special service type is used only to map a local service name to an external DNS name. It is useful when our containers want to reach some external service, but using a well-known local service name, that will not change instead of original external DNS name. Technically, Kubernetes will redirect our request to the original external location. Services of the ExternalName type does not support selectors, because there are no pods running behind.
The Ingress resource is a special object managing external access to multiple services in a Kubernetes cluster. It offers three major features: virtual hosting, load balancing and SSL termination.
Ingress gives us a single-entry point for our application composed of multiple Kubernetes services (called backend services here).
It routes incoming HTTP/HTTPS traffic based on information from a request URI: host name (e.g. sample.objectivity.co.uk) and path (e.g. /blogs/article/add). The routing rules are part of an Ingress definition. In addition, a default backend service can be specified that receives traffic not matching any defined routing rules. Currently, only HTTP/HTTPS protocols are supported by Ingress.
All external traffic to our application can be secured by using the HTTPS protocol and TLS encryption. Ingress simplifies this task, because it offers the TLS termination feature. To make it work, an Ingress definition must reference a Kubernetes secret with appropriate TLS certificate and private key. There are some useful tools, like cert-manager that runs in a cluster, automatically obtains free Let’s Encrypt certificates and associates them with an Ingress instance.
The below YAML example is a very simple Ingress definition. It configures secure HTTPS connection, where appropriate certificates are included in a tls-secret cluster resource.
There are two backend services connected using path routing rules. For example, requests starting with https://sample.objectivity.co.uk/service-a are handled by the my-service-a backend. There is also a default backend service my-default-service defined, that is run when no routing rule can be applied.
The Ingress resource is only an abstraction – it needs a 3rd party software implementing it, called Ingress controller. There are many available products we can choose from, like NGINX Ingress Controller, Ambassador, Voyager, Contour, HAProxy Ingress Controller, Traefik.
While the Ingress resource was primarily designed for facilitating external access to our Kubernetes services, it could be used also for internal communication between containers within a cluster. But in such a scenario we are limited only to HTTP/HTTPS protocols.
With the occurrence of complex microservices-based applications, running hundreds or thousands of specialized software services, a concept of service mesh was invented. Service mesh is usually defined as a special network layer (over TCP/IP) used for fast, reliable and secure communication between services (any separate processes connected with a network). Looking from the technical perspective, a service mesh can be understood as a set of connected network proxies. Each instance of our services is attached with such a sidecar proxy. The proxy’s primary aim is to intercept all network traffic to/from our service. What is really important is that the application code should not be aware of the proxies’ existence. The application should continue to use standard network communication methods, without any code changes needed for a service mesh adoption. A sidecar proxy is responsible for capturing a request, recognizing a target service instance and sending data to it. The below picture 3 gives a general overview of a service mesh.
Service mesh offers much more that simply delivering data from one service to another. It provides solutions for many cross-cutting concerns related to services communication, like:
- service discovery,
- security (encryption, authentication, authorization),
- fault tolerance (circuit breaker pattern, retry policy),
- load balancing.
In general, service mesh products are composed of two main parts. Data plane is responsible for realizing all network traffic between services (east-west traffic). Control plane allows to configure the traffic rules. The first proprietary service mesh solutions were developed many years ago by big companies like Google, Twitter or Netflix to face the inter-process communication problems in their growing microservices ecosystems. Later, open-source products supporting cloud native applications started to appear. Today there are two leading service mesh products available: Istio and Linkerd.
Istio, the service mesh implementation we used in one of our projects, is designed for use with the Kubernetes orchestrator only. Its network proxies are based on Envoy and deployed as sidecar containers within pods in a cluster. These proxies can capture HTTP, gRPC, WebSocket or TCP traffic from the primary application container in a pod and send it to appropriate target service instance.
The data plane in Istio is made of the mentioned Envoy proxies and the special Mixer component, responsible for telemetry and various policies enforcement (e.g. authorization, quotas, etc.). The Istio’s control plane contains three main components: Pilot (service discovery, routing, failure handling), Citadel (authentication, identities, keys and certificates) and Galley (service mesh configuration management). In addition, there are other useful 3rd party tools that can be integrated with an Istio service mesh and provide valuable insights into our cluster and inter-services communication. Good examples are Prometheus (metrics and monitoring), Kiali (observability console) or Jaeger (distributed tracing).
The Istio service mesh is still not a mature product, but its development is supported by companies like IBM, Google or Lyft, so we can expect it will gain popularity and importance. Service mesh products seem to be a great solution for handling inter-containers communication, because they separate this subject from the application code and allow to manage the communication in a very comprehensive and separate way. We do not need to implement a lot of common concerns. Of course, it all comes with the price of high configuration complexity.
There are multiple ways of communication between processes running as containers in a Kubernetes cluster.
Some of them were presented briefly in this article, but there could be other options. For instance, containerised applications can utilize message brokers (like Azure Service Bus or RabbitMQ) to interchange data in an asynchronous manner. However, the main purpose of the article was to describe more direct communication methods to be achieved inside a Kubernetes cluster. Implementing of communication between containers within a single pod should not be a common or often-performed task. The recommended approach is to put our containerised application inside a separate pod. But when we need to implement a specialised sidecar container supporting our main application, we can just use standard IPC methods, shared files or localhost networking.
The Kubernetes service resource should be used to expose our pods to other pods or even outside a cluster. A typical multi-services application would not need Ingress for internal communication between pods within a cluster (it is not the Ingress intended purpose). Instead, a set of Kubernetes services with internal cluster IPs only can communicate directly, using internal DNS names preferably.
The idea of a service mesh looks very promising. A lot of communication-related aspects can be covered by a single product like Istio, created for containerised services especially. It sounds good, but probably we need to wait for it to become more mature and easier to use.