Deploy Flink Jobs on Kubernetes

In case you need to refresh some basic concepts you can read my first story of Flink.

you: Hi, I need to change some configurations in Flink cluster, you have time to help me?

infra team: we don’t have time today and tomorrow we will proceed with resilience tests, maybe in two days we can schedule some time to discuss what you really want to do

infra team: Hi, regarding the talk we had three days ago, can you explain better what you want?

you: Hi! I need to change some configurations in the flink cluster (staging environment) to test bad behaving from S3 external dependency and validate if the job recoveries it self without losing information

infra team: I see. Maybe next week we can tackle that

Kubernetes

Pods are the smallest deployable units of computing that can be created and managed in Kubernetes.

A ReplicaSet’s purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.

An abstract way to expose an application running on a set of Pods as a network service.

A Deployment provides declarative updates for Pods and ReplicaSets.

Creating Flink Cluster on Kubernetes

Example 1. Kubernetes Cluster started using Minikube
minikube start — vm-driver=<driver_name>
minikube start — kubernetes-version <kubernetes_version>.
Deployment object for a JobManager
Deployment object for TaskManager
Service object for JobManager
kubectl create -f jobmanager.yaml
kubectl create -f jobmanager-service.yaml
kubectl create -f taskmanger.yaml
kubectl get deployments
The output of the command get deployments
kubectl get pods
The output of the command get pods

Hum ok, Job Parallelism is easy but Data Parallelism and Task Parallelism? Excuse me …

Build Flink Job

Job source code
nc -lk 9999
mvn clean package

Running our first Flink Job

minikube service jobmanager --url
The output of the command service URL
Apache Flink dashboard
Flink Job Uploaded
Flink Job running
nc -lk 9999
Sending messages to our job deployed

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store