Expose the JMX metrics in a Prometheus format At this point, the metric server custom-metrics-stackdriver-adapter feeds the Kubernetes master with the new custom metric values. Now lags can be plotted, but also queried by a metric server. Each application instance has a sidecar prometheus-to-sd to scrap the metrics and send them to Stackdriver. We first expose JMX metrics of the streaming application in Prometheus format. Among all possible implementation of adapters, we chose the adapters based on Stackdriver to create a bridge between Kubernetes custom metrics and our Kafka-Streams JMX metrics:įig 2 : Exporting metrics from Kafka-Streams This feature requires to enrich the original Kubernetes APIs with additional adapters. Fortunately, since version 1.6, it can also scale applications on custom metrics. It can also natively scale (in or out) based on CPU usage or memory consumption. Kubernetes maintains the number of deployments asked for a given application. This looks like a perfect job for Kubernetes! We have several instances of the same application, and we’d like to add or remove them based on the evolution of a shared metric exposed by the application: the consumer record-lag. We have covered a lot so far, let’s go through it again before moving on. The record-lag of the group corresponds to the sum of each instance lag.įig 1 : Kafka-Streams and the consumer-group Kubernetes and the custom metrics support All instances from a same consumer-group consume a distinct sequence of messages. Given an input topic with N partitions, any N+1 instance would be in stand by, waiting for another instance failure. Finally our maximum parallelism level corresponds to the number of partitions. All partitions get redistributed on the two instances. Messages with the same key are stored in the same partition.īy starting a second instance of our application we trigger a partition rebalance. Reminder : by default messages are distributed based on their key. When a single instance of a streaming application is running, all the input topic partitions get assigned to it. Like most other distributed systems, Kafka makes a really good usage of the maxim “ divide and conquer”, now let’s see how it works in practice.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |