Work-management company monday.com had a problem: critical production K8ssandra clusters sitting on the wrong Amazon Elastic Kubernetes Service (EKS) clusters.
K8ssandra is a Kubernetes operator for Cassandra, the Apache open source NoSQL database widely used by Fortune 500s for handling large data flows across servers without a single point of failure.
As monday.com changed infrastructure provisioning and management, it had a new approach to Helm chart management and moved away from Terraform and CDKTF for EKS cluster creation.
"To stay aligned with this new standard and avoid relying on infrastructure that would eventually be deprecated, we needed to migrate our K8ssandra clusters to new EKS clusters," said DevOps engineer Amit Mazor in a blog post.
For all the ease with which it manages data centres, K8ssandra did not have a solution for moving to entirely new EKS clusters. You can add a new data centre to an existing cluster, but if you are moving to a new EKS cluster, that won't help you create the first data centre.
It can be done though, said Mazor – with zero application downtime – if you recognise that unlike stateless workloads, where speed is important, working with stateful systems is about "control, automation, and failure handling."
Control plane pause
After setting up the new K8ssandra operator on the new clusters, the team paused the K8ssandra control plane (by scaling down k8ssandra-operator-cass-operator and ‘k8ssandra-operator) to avoid race conditions between old and new.
"When we tried to delete the old data centers at the end, we failed because the old operator tried to recreate them," said Mazor.
The setup briefly involved two K8ssandra cluster resources (configured to behave as one) spanning three data centres.
