What is our primary use case?
The main use case was using Cilium as a CNI for our Kubernetes cluster, so we used it for network policy and observability via Hubble, primarily for our CNI for the Kubernetes cluster.
My responsibility was to implement and set up a Kubernetes cluster, and one of the things we had to choose was CNI. We had several options, one of them was Cilium, because it is eBPF based and operates at a kernel level performance, giving us better performance. After we set up that with Cilium CLI, it was somewhat difficult to set up, but afterwards, we could solve problems with the observability we had with Hubble UI. I remember one time we had to find out what the problem was between two microservices we had, and we used Hubble to understand what the problem was.
What is most valuable?
I can speak about the way we use Hubble mostly because it was really different compared to what other tools I used before for observability at the network level. It gives us exactly what all the packets are using in this cluster and what happened between microservices. I remember the latency we had between two microservices, and we could solve it to understand the main debugging we could do for that. It was also better when I read about the internal level of Cilium that uses eBPF, not using IP tables, giving us better performance because we used a two regions cluster. It also gave us native support for L3, L4, L7 network policies, allowing us to manage the policies between microservices.
The thing that made Cilium different compared to other CNIs was that it is eBPF based, because mostly others used IP tables, and IP tables is not very fast compared to eBPF. The other thing is Hubble, in my opinion, gave us very good observability on the whole traffic in our cluster and provided us with a better debugging version we could use, because normally for debugging network problems, it is really hard.
First of all, we wanted to use Flannel for our development environment, and we can say Flannel was fine enough, but when we used Cilium, we understood we could obtain more internal debugging knowledge that we needed. We had more knowledge and a higher observability level of all the packets, which gave us better performance too, because it was an exchange company needing lower latency and fast transactions. After we changed the CNI from Flannel to Cilium, even in the dev environment, we got better performance, approximately 20% better, and because we used Prometheus monitoring, we had accurate data about our performance improvements.
What needs improvement?
Regarding features, I can speak about the docs, which were very good regarding deep networking and the internal workings of Cilium, but sometimes I think they could be better, specifically for best practices. Some parts need improvement in including best practices for each section, such as upgrading or implementing it, because I had some problems with setting up with Helm, and then I used Cilium CLI. Then I understood Cilium CLI is the same as using Helm, so that was beneficial, but the docs of Cilium could be better.
I think the assumption for the readers is that they have really good deep knowledge about networking, but sometimes, in my opinion, to share something with that good performance and those good tools, we need to make it simpler to teach others about eBPF. I read the book about eBPF shared on LinkedIn, and it was a good book, but we can make it simpler by sharing best practices and conducting interviews with users about how they utilize that, making it easier for everyone to start and understand. Improving the documents is important.
I remember when I set up the Hubble UI, there was a bug, but it is already fixed because when I updated, it was resolved. The bug was when I implemented and set up Cilium CNI, Hubble could not run, but after I upgraded the Cilium version, it was fixed. For improvements, having some tools for debugging eBPF at the kernel level could help. I know we have those, and I use them, but a better UI for that would be beneficial. Also, the documentation can be better and more straightforward, perhaps offering two versions—for those who want to learn more and for those who just want to implement with less focus on the deeper networking aspects.
I would rate Isovalent an eight or nine because almost everything is fine, with very good quality and support in the open-source community. However, as I said, the documents can be improved, and as a DevOps engineer, I always look for best practices to set up or configure something easily. While best practices are environment-based, there are some default best practices that could be highlighted somewhere to make it much easier for others to set up these tools.
For how long have I used the solution?
In both companies I worked for, Invex and Norvan, I used Cilium, the CNI for our Kubernetes cluster, so I have worked with that for approximately two years.
What do I think about the stability of the solution?
In my opinion, during the two years we used it, we did not have many problems with Cilium. There was an issue with the initial setup on-premises, but once configured well, it operates very smoothly with minimal problems.
What do I think about the scalability of the solution?
The scalability is very good.
We handled two clusters in different regions with many nodes, around 20, facilitating connections between those nodes and the pods, which numbered more than 200. Regarding customer support, since I used the open-source version, not the enterprise version, it was very fine. I remember asking questions in Slack, and the community was active, responding in a decent time.
How are customer service and support?
Since I used the open-source version, not the enterprise version, it was very fine. I remember asking questions in Slack, and the community was active, responding in a decent time.
Which solution did I use previously and why did I switch?
We did not use Flannel for long but made a short shift to Cilium in the dev environment. In production, we used Calico for some months, but we switched to Cilium due to performance benefits, as we wanted to focus on understanding how to choose something based on future needs to outperform our competitors in the market.
How was the initial setup?
My responsibility was to implement and set up a Kubernetes cluster, and one of the things we had to choose was CNI. We had several options, one of them was Cilium, because it is eBPF based and operates at a kernel level performance, giving us better performance. After we set up that with Cilium CLI, it was somewhat difficult to set up, but afterwards, we could solve problems with the observability we had with Hubble UI. I remember one time we had to find out what the problem was between two microservices we had, and we used Hubble to understand what the problem was.
Which other solutions did I evaluate?
We evaluated Flannel, Cilium, and Calico, and based on those experiences, we chose Cilium due to its superior performance and better observability provided by the Hubble tool.
If you have a good product needing high performance and visibility into your network at the cluster level, I suggest giving Cilium a chance. You will understand the great observability of Hubble and the performance benefits of eBPF, especially when you compare it to IP tables and see how it reduces the complexities of IP table rules.
What other advice do I have?
The open-source community is active and robust.
If you have a good product needing high performance and visibility into your network at the cluster level, I suggest giving Cilium a chance. You will understand the great observability of Hubble and the performance benefits of eBPF, especially when you compare it to IP tables and see how it reduces the complexities of IP table rules. I would rate this review a nine overall.
Which deployment model are you using for this solution?
Hybrid Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?