![]() ![]() When your Kafka broker is scheduled to a node, it can use the ephemeral volume and store data in it.īut when the node crashes or the Pod is rescheduled, it will receive a new ephemeral disk which will not contain the previously stored data. It behaves very similarly to a temporary directory. The ephemeral storage behaves differently.Īn ephemeral volume is usually just a directory somewhere in the host operating system of your node. It has to resync only the records which it missed during the failover. ![]() It will need to resync the partitions which it hosts, but it doesn’t need to start from scratch. So when the broker starts, it will find the same data which it used before. The persistent volume with the data used by the given broker will be reattached to the new node as well. This usually takes between several seconds and several minutes, depending on your cloud provider and storage implementation. So when the node where your Kafka broker is running suddenly crashes, Kubernetes will be able to reallocate your broker to a different node in your cluster. The persistent network attached storage is not tied to any particular node. Kubernetes for a long time supported two kinds of storage:Įach behaves a bit differently when your Pods with kafka brokers are rescheduled or when the node they are running on fails. You need to use the storage as supported by your platform. ![]() In fact, whatever storage is supported by your operating system will work.īut what if you are running Kafka on Kubernetes or on OpenShift platforms? When running Kafka on bare metal or on virtual machines, both local as well as network attached storage are quite easy to use. This can be done easily even with the (often much cheaper) local storage. Topic replication can be used to achieve high availablity and data durability without expensive storage.Īll Kafka records will be shared between several nodes and when one of them crashes or dies, another can take over. In fact, a lot of people would tell you that they prefer to use Kafka with local storage. It is also easy to reattach to another machine when needed (for example because the original machine died).īut Network Attached Storage has also some drawbacks.įor example, to achive good and reliable performance it requires a very good network.īut Kafka doesn’t really need network attached storage. It often promises higher reliability and great performance. When deploying new Kafka cluster, one of the first decisions you have to make is the type of storage you want to use.Įven before deciding on such storage aspects such as the type of filesystem which should be used, you have to decide a more basic question: should you run Kafka with local storage or with network attached storage? Local versus network storageįor a lot of stateful applications, Network Attached Storage (NAS) is often seen as the prefered choice. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |