One of the features I find valuable in PagerDuty Operations Cloud, which is part of our current migration activities, involves automating the entire incident management process by integrating all service incidents into a single incident management page. In IBM cloud, since many services and incidents occur, I utilize the runbook automation feature where I create runbooks for each service and common issues that facilitate incident management for common incidents. The runbook automation positively impacts my team's workflow by significantly speeding up the incident resolution process. In IBM cloud, we have different services hosted such as IBM Schematics and IBM Kubernetes Service, with thousands of concurrent global users. We face several issues during multiple incidents, particularly in reliability and infrastructure side issues. Basic level zero incidents often require simple commands run in kubectl. Therefore, I created runbooks to address these common issues, allowing SRE team members to refer to the runbook and manually fix the issues. However, before using PagerDuty Operations Cloud's runbook automation, it took over 20 minutes to resolve these issues. After implementing PagerDuty Operations Cloud's runbook automation, I have reduced the response time from over 20 minutes to less than two minutes, saving approximately 80 to 90 percent of the time and making mean time to resolve significantly faster. I review the runbooks quarterly to update them with any new steps necessary. PagerDuty Operations Cloud has greatly improved our productivity. Previously, I handled many incidents with manual automation runbooks, leading to substantial toil for the SRE teams in resolving even minor incidents and complicating our on-call schedule. Once I adopted PagerDuty Operations Cloud and heavily utilized the runbook automation, I provided a list of common incidents to PagerDuty Operations Cloud, which then did the heavy lifting in fixing basic incidents automatically. This allowed my team to focus more on development activities related to platform engineering. Overall, it has reduced our toil by at least 50 to 60 percent and improved our efficiency, enabling us to onboard more services. We increased from onboarding seven core IBM cloud services to over 28 services now hosted. The expansion of services impacts our organization's goals and customer experience by allowing all IBM cloud internal services to be hosted on a dedicated platform engineering service called Rednote. Before using PagerDuty Operations Cloud, I utilized Nagios and Sysdig, and after migrating to PagerDuty Operations Cloud last year, I prepared a set of runbooks, automating on-call schedules and incident management. This automation has led to a significant increase in incident closure rates from around 40 to 45 percent, improving efficiency and reducing manual toil for basic incidents by about 60 percent. This enables my team to focus more on development activities as basic incidents that can be managed through simpler runbooks are now handled automatically by PagerDuty Operations Cloud. Additionally, incorporating AIOps into our on-call scheduling and notifications helps it learn from previous incidents and proactively address issues. This scalability has allowed me to grow from handling four cloud services to 28 services, increasing from 200 to 300 customers to over 1800 plus customers, thanks to PagerDuty Operations Cloud.
