What is our primary use case?
I'm a DevOps Architect for our on-premises multi-cloud solution. The core business of the company I work for is clearing between all the banks in my country.
We're now in the phase of implementation of major projects for Instant Payments and Open Banking API HUB (PSD2). One of our requirements from solution providers was that the solutions should be "Kubernetes agnostic", so we can deploy the solution on OpenShift, vanilla Kubernetes, etc.
At the same time, for all newly developed in-house applications, we strongly prefer to be able to be deployed on Kubernetes too.
Since we're a small company we need to be efficient and that is why we are implementing automated releases, automatic testing, and CI/CD pipelines using seamless microservices.
When I started using OpenShift, it proved to be a very resilient platform with strengthened security. My responsibility was to build up the whole on-premise private multi-cloud. We designed a practice in GitOps manner so that the source code of all the applications, including the manifests for Kubernetes clusters and the database scripts management, are stored on our private GitHub repository that is used as a single source of truth. At the same time, we realized, that we cannot store sensitive data in GitHub like credentials to other systems, certificates, API keys, etc.
To solve this, we deployed and configured another system place for the secure storage of sensitive data - HashiCorp Vault.
Since we have two data centers, we decided to do multi-cloud deployments in a manner that we don't have one OpenShift cluster stretched across different data centers, but for each data center, we have a separate deployment where both of the OpenShift clusters are connected to the same database and the same external persistent storage. The production databases are not deployed on OpenShift.
We heavily use cloud-native CI/CD pipelines. Practically, there are two cloud-native projects that are supported by RedHat. Tekton is used for Continuous Integration and ArgoCD is used for Continuous Deployment.
Red Hat developed operators for these two tools and we embedded those tools in our day-to-day work. Whenever a developer performs a specific action on GitHub, like Pull Request, separate processes automatically start through the use of webhooks. Then, that source code is built, tested, packaged into a container, pushed to our private docker registry, and deployed to various environments depending on the GitHub branch (dev, test, prod).
We created generic CI/CD pipelines for APIs and databases and by using these pipelines we're speeding up the process of day-to-day deployments, like manually copying files and assigning privileges to various environments. Now, the rest of the staff has more time for more concrete jobs rather than repetitive tasks that are also very prone to errors. We're much more efficient now.
Our developers aren't using CodeReady workspaces and they don't have isolated environments on their laptops, but they do have access to the OpenShift clusters in a very controlled manner since they're directly deploying the solution via these pipelines to the OpenShift platform.
We're currently using version 4.10. I have also used versions 3.10 and 3.11. We are testing deployment on OpenShift 4.12 too where the Kubernetes version is 1.25+ which brought a lot of major improvements.
How has it helped my organization?
When there are a handful of APIs and microservices, you can orchestrate them manually and use Docker Swarm. However, when there are dozens of applications consisting of hundreds of microservices, you must use tools for orchestration. That's where Kubernetes comes into play.
When I came to this company all of the deployments to various environments (dev, test, prod) were done manually.
As I said, there are two main types of deployments. One type is the deployment of databases, DB tables, new procedures, new functions, etc.
The other type is the deployment of APIs. For that deployment, my colleagues were manually building the project. They did not use any kind of SCM tool at that moment. When they got the compiled artifacts, like DLL files, they were copying those files to another team. People from that team were picking the files and manually copying them to our web servers.
For databases, it was practically the same. Developers built escrow scripts, and another team deployed the scripts and tested them on a test environment. After successful testing, the third team deployed the scripts to production.
This was a very manual, lengthy, and error-prone process where a lot of things could go wrong.
We now use our private GitHub repository for software development and version control, as well as CI/CD pipelines for both API and database deployment to each environment.
It's a very automatic and predictable process.
Deployment of a new release to the test or production environment is protected, and developers can't deploy a new version any time they want. They need to have approval from the release manager. If the release manager approves the deployment to a specific environment, the pipeline automatically picks up that information and deploys the whole solution.
This control of deployment to a specific environment (four eyes principle) is managed with policies on GitHub.
We have reports in the form of audit logs that show who did what, at what time, and in what environment and we have total control over the production environments.
We have achieved zero downtime on almost all levels, because of the way that OpenShift and the containers work. When a new container is deployed, the old one is still running. When the new one starts and all of the readiness probes work, then the old container is practically terminated. Most of the time, we don't have downtime for some of the manual work.
Using GitHub and CI/CD pipelines, we have avoided a lot of manual work and have sped up the process of deployment of new releases to various environments by at least 30%. Now, we have a more reliable process, which is automatic, and we're avoiding human-prone mistakes during this process.
OpenShift also comes with integrated monitoring. It uses internal metrics collected by Prometheus and Grafana for visualization of those metrics, but you can export all the logs and metrics from OpenShift to another system. You can also transfer the logs and metrics to Elastic Stack using filebeats, metricbeats, and auditbeats, and you can easily monitor the cluster from outside because you need to be on top of it all the time.
When everything goes down, you must intervene quickly and get information very quickly through different channels like Slack, SMS, email, Teams, or chat.
What is most valuable?
Security is the most valuable feature. If you get Vanilla Kubernetes, they lack security. Red Hat OpenShift comes in two flavors. One is OCP, OpenShift Container Platform, for which you need licenses. We're using that for production environments. For developing environments, there is the OKD Community edition of OpenShift.
They're very similar because OCP uses Red Hat CoreOS certified from Red Hat, but the community edition uses Fedora CoreOS. We're trying to deploy applications to be “Kubernetes agnostic” about the underlying infrastructure. Whatever we deploy should work on both OCP and OKD. In that pattern, we're also practically saving on licenses because we use them for the production version of OpenShift only. For development and testing, we use OKD.
Three years ago, deployment of the OpenShift cluster wasn't easy. It required a lot of knowledge of load balancers, networking, DNS and DHCP services, and virtualization. For version 3.11, OpenShift came with Docker as a container's run-time engine. From version 4, Docker was replaced with Podman, which is quite a good approach because Docker needs to run as a daemon with elevated privileges. Podman doesn't require such elevated privileges.
In the beginning, it was very difficult to install OpenShift even by following the documentation. There were some YouTube videos, but we struggled. That installation was named UPI, which stands for user provision infrastructure. That means that you need to deploy your own load balancers to configure them correctly and enter your DNS and domains. Only if everything is configured correctly, the OpenShift cluster will work.
Then, Red Hat came up with a solution. We use virtualization technologies on-premises. We do not use bare metal, so this was a very hard task on VMware, but then Red Hat from version 4.5. updated their installer to use IPI (installation provisioned infrastructure). For day-to-day jobs, we prepared one helper machine from which we can manage, deploy, destroy, and operate multiple clusters from one place. The bare installation of deploying an OpenShift cluster is now an easy task for us.
The stack in the software supply chain is one of the main reasons that we use OpenShift. When I came to this company, we bought hardware from IBM named Bluemix, and they used ICP, which stands for IBM Cloud Private.
Today, you can have Kubernetes on IBM, Amazon, Microsoft, Google, name it. You can also have different installations on various platforms like VMware and Tanzu, which are commercial products. Also, there are some open-source variations of Kubernetes like Rancher and Platform9.
At that point in time, IBM bought Red Hat. They very cleverly recognized that their product, IBM Cloud Private, was an inferior platform to Red Hat and OpenShift, and they invested a lot in Red Hat and in OpenShift. OpenShift is an enterprise-grade standard Kubernetes orchestration system for huge enterprises. There are more flavors of Kubernetes, but I believe OpenShift is practically a standard one, so whether or not you use it on the cloud or on-premises, OpenShift has a huge market share.
Vanilla Kubernetes lacks security. We have role-based access controls to tune and perform grant-level access to specific service accounts, roles and permissions. There is very good isolation between the namespaces. Practically, we have four clusters on-premises for each huge specific production grade system. Two of those clusters run several independent environments that are different from each other on the same clusters (dev, performance, smoke, test). Only production clusters are separate from those. No deployment sees another deployment, so they cannot interfere.
We connected our OpenShift platform with our LDAP, so we also have security that shows who accesses it and what permissions they can perform over the operative clusters and applications that run on them. We have a DevOps team, developers, and infrastructure guys, and there haven't been any complaints so far about the day-to-day usage of OpenShift. I believe all the aspects of security that we need are practically covered from both the user perspective and the application perspective.
What needs improvement?
OpenShift has a pretty steep learning curve. It's not an easy tool to use. It's not only OpenShift but Kubernetes itself. The good thing is that Red Hat provides specific targeted training. There are five or six pieces of training where you can get certifications. The licenses for OpenShift are pretty expensive, so they could be cheaper because the competition isn't sleeping, and Red Hat must take that into account.
There are a few versions of OpenShift. There is the normal OpenShift and an OpenShift Plus license. Red Hat could think of how to connect those two subscriptions because, with Red Hat Plus, you have one tool called ACM (Advanced Cluster Management), where you can manage multiple clusters from one place. We deployed this functionality by ourselves, but if you don't pay the license for Red Hat OpenShift Plus, you'll lack this functionality. If you have a multi-cloud environment and you have a lot of work to do, it would be a plus if the Red Had OpenShift Plus license came in a bundle with the regular solutions. This ACM tool should be available in the normal subscription, not just the Plus version.
There are new versions on an almost weekly basis. I found myself that the upgrading of OpenShift clusters is not a task that will successfully finish every time. It's a simple and quick, but not reliable process.
That's why we use multiple clusters. We use v4.10.3, but we want to move to v4.12.X. The upgrade process itself can fail, and we don't have backups of our OpenShift cluster because we have backups of all the Kubernetes manifests on GitHub.
We destroy the cluster, bring up a new one quickly, and apply those scripts. The upgrade itself could be more resilient for us as administrators of OpenShift to be sure that it'll succeed and not occasionally fail.
They can improve the reliability of their upgrade process. They also have implementations of some Red Hat-verified operators for a lot of products like Elasticsearch. They're good enough for development purposes, but some of the OpenShift operators still lack resilient production-grade configurations.
Red Hat says that we have a few hundred operators, but I believe that only half of them are production-grade ready at this moment. They need to work much more on those operators to become more flexible because you can deploy all of them in development mode, but when we go to production grade and want to make specific changes to the operator and configuration, we lack those possibilities.
For how long have I used the solution?
I have used Red Hat OpenShift for three years.
What do I think about the stability of the solution?
There are some undocumented features and well-known bugs, and Red Hat is aware of them. I don't know why they haven't been fixed so far. Otherwise, it's a stable product. We use it in production for critical areas, since we learned how to avoid some of the known bugs. Bugzilla is a tool we use to report some of the bugs to RedHat Support.
We usually have initial scripts where we deploy the solution for a new cluster and eliminate or manually fix some of those issues. It's a stable product, but Kubernetes is a platform for making platforms. It gives us the flexibility to easily speed up our process and not waste manual work. It can be improved in many ways, but that will take some time.
What do I think about the scalability of the solution?
The scalability is the main reason why we use OpenShift. We heavily use automatic scalability because we use horizontal and vertical pod auto scalers. When we configure our OpenShift clusters in a way where we defined the required resources for each and every pod and when that pod comes to 80% of CPU or memory usage, the system automatically brings up a new pod.
It can scale up to a certain amount of pods and if there's very heavy utilization of the OpenShift cluster itself, they OpenShift automatically provisions a new worker node for you. We tested it and it works.
When it's required that the system isn't under heavy load, those nodes can be destroyed automatically. It's a very nice feature compared to the monolithic applications and automatic provisioning of the new virtual machine installation and all of the configurations and operating systems. The scalability and orchestration of many containers is the main reason why we use OpenShift.
How are customer service and support?
We contacted support a few times, but it wasn't great. It could be improved a lot.
We contacted them about licensing. It wasn't clear whether it was being done on CPU or an entitlement level because we bought our product from IBM, but IBM bought Red Hat. Still, we struggle with the support because we report our complaints to IBM first, and then IBM redirects us to Red Hat.
About four years ago, IBM bought Red Hat. They still don't have one unified ticket system. You have to buy licenses from Red Hat separately. Although you can buy them from IBM, you need to import those licenses for Red Hat on a separate portal. It requires some overhead from us, which isn't good. There were a few times when we contacted them, and they were quite efficient, but still not enough for us to wait for the OpenShift cluster to be fixed, so we redeployed the cluster.
I would rate technical support a five out of ten.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
Previously we had mainly monolithic applications and a fair number of APIs running on IIS. In order to be more efficient and resilient we switched to containerized APIs running on OpenShift Kubernetes.
How was the initial setup?
When we initially used ICP IBM private cloud, that platform used something called patterns. We had a pattern for the deployment of OpenShift. At that point in time, we used OpenShift 3.11. In the background, those patterns were practically Ansible scripts simulating UPI installation as IPI. Ansible is a Red Hat product for automatic infrastructure and configuration deployment.
At that time, we had user provision infrastructure, but instead of doing everything manually like provisioning load balancers, configuring them, and putting in the DNS records, those patterns were doing that for us. They were somehow simulating IPA, installation provision infrastructure. Then, we purchased a VxRail platform from Dell, and from there we can do everything on our own.
Starting from OpenShift v4.5. Red Hat, came with IPA installation for VSphere, so we can deploy OpenShift clusters quickly and efficiently on VMware with a few lines of code and a few YAML configuration changes. In the beginning, it was a hard task, but now once you prepare everything, it’s a quick and easy task.
Setup depends on how well you're prepared. I prepared a small helper machine and configured it with a lot of tools, and I manage all my clusters from that machine. It has a connection to my GitHub. I have all of my repositories and Kubernetes manifest for various deployments stored on that machine too.
If you're deploying OpenShift for the first time, it can take several hours if everything goes well. When we first performed all of these prerequisites, we imported the VMware certificates and performed all the configurations, and we only needed to change very small details for the new OpenShift cluster. It's usually a few commands, and the installation itself takes approximately 40-60 minutes.
After initial installation, it takes approximately another 30-40 minutes to configure a freshly deployed OpenShift cluster. In about hour and a half, I can deploy any cluster with all the applications and configurations running, assuming that we have everything prepared, and stored on GitHub.
As I said, we rarely contact Red Hat support because we use GitOps practices. For High Availability, we always have at least two clusters in two different data centers and in case we cannot fix the cluster itself within 30 minutes we recreate it from scratch. In front of the OpenShift clusters there are few clustered load balancers. If one cluster is down, the traffic goes to another cluster.
What other advice do I have?
I would rate OpenShift an eight out of ten.
My advice to new users is to try OpenShift Container Distribution. You can try to deploy some applications from source code or a Docker container. That product is free, so you don't need to pay anything. If you want to try the real thing, Red Hat offers a 60-day trial so you can install Red Hat OpenShift on-premises on a virtualization platform or on bare metal.
After each trial period, if no license is activated in the cluster you are losing the ability for upgrading the cluster. The good thing is that anybody can try it, but it's very different if you use OpenShift in the cloud or on-premises. On-premises, you can use whatever you want. You can use a few laptops and desktops or you can have a virtualization platform, but you should always check the documentation and see whether that platform is supported.
The first steps are always hard. You must go through the documentation and learn how to set it up and how to make the first installation. When you make the first installation, it's a lot easier, but it's not only the installation of OpenShift itself. It's the Day-2 Operation activities, how to monitor the OpenShift cluster, how to deploy workloads, and how to automate the workload deployment. It's a steep learning curve, but we have seen great satisfaction from our colleagues. It's good to invest in this knowledge.
Red Hat has OpenShift.TV, which is a YouTube channel where they discuss new ideas, tips, and tricks once or twice a week. I contacted a few Red Hat ambassadors that I saw in those YouTube videos. I was surprised when I received a response from them within 24 hours. It helped me a lot.
Which deployment model are you using for this solution?
On-premises