This post was originally published on the ETI blog here.
Today I was given the challenge of providing Kafka as a service to multiple development teams in a way that was consistent and could be managed easily. There are a number of challenges to this, from how do you provision the service request through to when the thing is running, how does it get monitored or upgraded.
Kafka is a streaming tool designed to be a highly available and scalable platform for building pipelines for your data and is used by many companies in production.
I wanted to deploy the ability to manage Kafka centrally, so an operator deployed once, centrally to provide Kafka as a service to development teams was a natural fit. It means that developers are able to quickly service their own needs and the central Cloud team stays off their critical path and can focus on providing platform features, not servicing individual requests.
The cleanest way to provide this type of centrally managed service is to deploy Kafka using an operator. Even though operators are only recently starting to be adopted, I was not disappointed to discover that the Strimzi project gives us a way to do this. I won’t cover what operators are in this article, but if you’d like to find out more about them, take a look at this blog post. There is also a set of training scenarios available on katacoda.
